Chapter 3. Deploying OpenShift sandboxed containers on AWS You can deploy OpenShift sandboxed containers on AWS Cloud Computing Services by using the OpenShift Container Platform web console or the command line interface (CLI). OpenShift sandboxed containers deploys peer pods. The peer pod design circumvents the need for nested virtualization. For more information, see peer pod and Peer pods technical deep dive . Cluster requirements You have installed Red Hat OpenShift Container Platform 4.14 or later on the cluster where you are installing the OpenShift sandboxed containers Operator. Your cluster has at least one worker node. 3.1. Peer pod resource requirements You must ensure that your cluster has sufficient resources. Peer pod virtual machines (VMs) require resources in two locations: The worker node. The worker node stores metadata, Kata shim resources ( containerd-shim-kata-v2 ), remote-hypervisor resources ( cloud-api-adaptor ), and the tunnel setup between the worker nodes and the peer pod VM. The cloud instance. This is the actual peer pod VM running in the cloud. The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass ( kata-remote ) definition used for creating peer pods. The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit attribute in the peerpodConfig custom resource (CR). The peerpodConfig CR, named peerpodconfig-openshift , is created when you create the kataConfig CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator namespace. The following peerpodConfig CR example displays the default spec values: apiVersion: confidentialcontainers.org/v1alpha1 kind: PeerPodConfig metadata: name: peerpodconfig-openshift namespace: openshift-sandboxed-containers-operator spec: cloudSecretName: peer-pods-secret configMapName: peer-pods-cm limit: "10" 1 nodeSelector: node-role.kubernetes.io/kata-oc: "" 1 The default limit is 10 VMs per node. The extended resource is named kata.peerpods.io/vm , and enables the Kubernetes scheduler to handle capacity tracking and accounting. You can edit the limit per node based on the requirements for your environment after you install the OpenShift sandboxed containers Operator. A mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available. The mutating webhook modifies a Kubernetes pod as follows: The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIME_CLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIME_CLASS , the webhook exits without modifying the pod. If the RuntimeClassName values match, the webhook makes the following changes to the pod spec: The webhook removes every resource specification from the resources field of all containers and init containers in the pod. The webhook adds the extended resource ( kata.peerpods.io/vm ) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes. Note The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource. As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces. 3.2. Deploying OpenShift sandboxed containers by using the web console You can deploy OpenShift sandboxed containers on AWS by using the OpenShift Container Platform web console to perform the following tasks: Install the OpenShift sandboxed containers Operator. Enable ports 15150 and 9000 to allow internal communication with peer pods. Create the peer pods secret. Create the peer pods config map. Create the KataConfig custom resource. Configure the OpenShift sandboxed containers workload objects. 3.2.1. Installing the OpenShift sandboxed containers Operator You can install the OpenShift sandboxed containers Operator by using the OpenShift Container Platform web console. Prerequisites You have access to the cluster as a user with the cluster-admin role. Procedure In the web console, navigate to Operators OperatorHub . In the Filter by keyword field, type OpenShift sandboxed containers . Select the OpenShift sandboxed containers Operator tile and click Install . On the Install Operator page, select stable from the list of available Update Channel options. Verify that Operator recommended Namespace is selected for Installed Namespace . This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created. Note Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail. Verify that Automatic is selected for Approval Strategy . Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available. Click Install . Navigate to Operators Installed Operators to verify that the Operator is installed. Additional resources Using Operator Lifecycle Manager on restricted networks . Configuring proxy support in Operator Lifecycle Manager for disconnected environments. 3.2.2. Enabling ports for AWS You must enable ports 15150 and 9000 to allow internal communication with peer pods running on AWS. Prerequisites You have installed the OpenShift sandboxed containers Operator. You have installed the AWS command line tool. You have access to the cluster as a user with the cluster-admin role. Procedure Log in to your OpenShift Container Platform cluster and retrieve the instance ID: USD INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' \ -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g') Retrieve the AWS region: USD AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') Retrieve the security group IDs and store them in an array: USD AWS_SG_IDS=(USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} \ --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \ --output text --region USDAWS_REGION)) For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel: USD for AWS_SG_ID in "USD{AWS_SG_IDS[@]}"; do \ aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 15150 --source-group USDAWS_SG_ID --region USDAWS_REGION \ aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 9000 --source-group USDAWS_SG_ID --region USDAWS_REGION \ done The ports are now enabled. 3.2.3. Creating the peer pods secret You must create the peer pods secret for OpenShift sandboxed containers. The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials. Prerequisites You have the following values generated by using the AWS console: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY Procedure In the OpenShift Container Platform web console, navigate to Operators Installed Operators . Click the OpenShift sandboxed containers Operator tile. Click the Import icon ( + ) on the top right corner. In the Import YAML window, paste the following YAML manifest: apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2 1 Specify the AWS_ACCESS_KEY_ID value. 2 Specify the AWS_SECRET_ACCESS_KEY value. Click Save to apply the changes. Navigate to Workloads Secrets to verify the peer pods secret. 3.2.4. Creating the peer pods config map You must create the peer pods config map for OpenShift sandboxed containers. Prerequisites You have your Amazon Machine Image (AMI) ID if you are not using the default AMI ID based on your cluster credentials. Procedure Obtain the following values from your AWS instance: Retrieve and record the instance ID: USD INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g') This is used to retrieve other values for the secret object. Retrieve and record the AWS region: USD AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"USDAWS_REGION\"" Retrieve and record the AWS subnet ID: USD AWS_SUBNET_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region USD{AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"USDAWS_SUBNET_ID\"" Retrieve and record the AWS VPC ID: USD AWS_VPC_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region USD{AWS_REGION} --output text) && echo "AWS_VPC_ID: \"USDAWS_VPC_ID\"" Retrieve and record the AWS security group IDs: USD AWS_SG_IDS=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region USDAWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"USDAWS_SG_IDS\"" In the OpenShift Container Platform web console, navigate to Operators Installed Operators . Select the OpenShift sandboxed containers Operator from the list of operators. Click the Import icon ( + ) in the top right corner. In the Import YAML window, paste the following YAML manifest: apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7 DISABLECVM: "true" 1 Defines the default instance type that is used when a type is not defined in the workload. 2 Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads. 3 Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID. 4 Specify the AWS_REGION value you retrieved. 5 Specify the AWS_SUBNET_ID value you retrieved. 6 Specify the AWS_VPC_ID value you retrieved. 7 Specify the AWS_SG_IDS value you retrieved. Click Save to apply the changes. Navigate to Workloads ConfigMaps to view the new config map. 3.2.5. Creating the KataConfig custom resource You must create the KataConfig custom resource (CR) to install kata-remote as a RuntimeClass on your worker nodes. The kata-remote runtime class is installed on all worker nodes by default. If you want to install kata-remote on specific nodes, you can add labels to those nodes and then define the label in the KataConfig CR. OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime. Important Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors might increase the reboot time: A larger OpenShift Container Platform deployment with a greater number of worker nodes. Activation of the BIOS and Diagnostics utility. Deployment on a hard disk drive rather than an SSD. Deployment on physical nodes such as bare metal, rather than on virtual nodes. A slow CPU and network. Prerequisites You have access to the cluster as a user with the cluster-admin role. Optional: You have installed the Node Feature Discovery Operator if you want to enable node eligibility checks. Procedure In the OpenShift Container Platform web console, navigate to Operators Installed Operators . Select the OpenShift sandboxed containers Operator. On the KataConfig tab, click Create KataConfig . Enter the following details: Name : Optional: The default name is example-kataconfig . Labels : Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair. enablePeerPods : Select for public cloud, IBM Z(R), and IBM(R) LinuxONE deployments. kataConfigPoolSelector . Optional: To install kata-remote on selected nodes, add a match expression for the labels on the selected nodes: Expand the kataConfigPoolSelector area. In the kataConfigPoolSelector area, expand matchExpressions . This is a list of label selector requirements. Click Add matchExpressions . In the Key field, enter the label key the selector applies to. In the Operator field, enter the key's relationship to the label values. Valid operators are In , NotIn , Exists , and DoesNotExist . Expand the Values area and then click Add value . In the Value field, enter true or false for key label value. logLevel : Define the level of log data retrieved for nodes with the kata-remote runtime class. Click Create . The KataConfig CR is created and installs the kata-remote runtime class on the worker nodes. Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation. Verification On the KataConfig tab, click the KataConfig CR to view its details. Click the YAML tab to view the status stanza. The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata-remote installation. A message appears each time there is an update. Click Reload to refresh the YAML. When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata-remote is installed on the cluster. Additional resources KataConfig status messages Verifying the pod VM image After kata-remote is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider. Procedure Navigate to Workloads ConfigMaps . Click the provider config map to view its details. Click the YAML tab. Check the status stanza of the YAML file. If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully. Troubleshooting Retrieve the events log by running the following command: USD oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation Retrieve the job log by running the following command: USD oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs. 3.2.6. Configuring workload objects You must configure OpenShift sandboxed containers workload objects by setting kata-remote as the runtime class for the following pod-templated objects: Pod objects ReplicaSet objects ReplicationController objects StatefulSet objects Deployment objects DeploymentConfig objects Important Do not deploy workloads in an Operator namespace. Create a dedicated namespace for these resources. You can define whether the workload should be deployed using the default instance type, which you defined in the config map, by adding an annotation to the YAML file. If you do not want to define the instance type manually, you can add an annotation to use an automatic instance type, based on the memory available. Prerequisites You have created the KataConfig custom resource (CR). Procedure In the OpenShift Container Platform web console, navigate to Workloads workload type, for example, Pods . On the workload type page, click an object to view its details. Click the YAML tab. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example: apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ... Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type: To use a manually defined instance type, add the following annotation: apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1 # ... 1 Specify the instance type that you defined in the config map. To use an automatic instance type, add the following annotations: apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ... Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available. Click Save to apply the changes. OpenShift Container Platform creates the workload object and begins scheduling it. Verification Inspect the spec.runtimeClassName field of a pod-templated object. If the value is kata-remote , then the workload is running on OpenShift sandboxed containers, using peer pods. 3.3. Deploying OpenShift sandboxed containers by using the command line You can deploy OpenShift sandboxed containers on AWS by using the command line interface (CLI) to perform the following tasks: Install the OpenShift sandboxed containers Operator. Optional: Change the number of virtual machines running on each worker node. Enable ports 15150 and 9000 to allow internal communication with peer pods. Create the peer pods secret. Create the peer pods config map. Create the KataConfig custom resource. Configure the OpenShift sandboxed containers workload objects. 3.3.1. Installing the OpenShift sandboxed containers Operator You can install the OpenShift sandboxed containers Operator by using the CLI. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Create an osc-namespace.yaml manifest file: apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator Create the namespace by running the following command: USD oc apply -f osc-namespace.yaml Create an osc-operatorgroup.yaml manifest file: apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: sandboxed-containers-operator-group namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator Create the operator group by running the following command: USD oc apply -f osc-operatorgroup.yaml Create an osc-subscription.yaml manifest file: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.8.1 Create the subscription by running the following command: USD oc apply -f osc-subscription.yaml Verify that the Operator is correctly installed by running the following command: USD oc get csv -n openshift-sandboxed-containers-operator This command can take several minutes to complete. Watch the process by running the following command: USD watch oc get csv -n openshift-sandboxed-containers-operator Example output NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.8.1 1.7.0 Succeeded Additional resources Using Operator Lifecycle Manager on restricted networks . Configuring proxy support in Operator Lifecycle Manager for disconnected environments. 3.3.2. Modifying the number of peer pod VMs per node You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig custom resource (CR). Procedure Check the current limit by running the following command: USD oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}' Modify the limit attribute of the peerpodConfig CR by running the following command: USD oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1 1 Replace <value> with the limit you want to define. 3.3.3. Enabling ports for AWS You must enable ports 15150 and 9000 to allow internal communication with peer pods running on AWS. Prerequisites You have installed the OpenShift sandboxed containers Operator. You have installed the AWS command line tool. You have access to the cluster as a user with the cluster-admin role. Procedure Log in to your OpenShift Container Platform cluster and retrieve the instance ID: USD INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' \ -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g') Retrieve the AWS region: USD AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') Retrieve the security group IDs and store them in an array: USD AWS_SG_IDS=(USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} \ --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \ --output text --region USDAWS_REGION)) For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel: USD for AWS_SG_ID in "USD{AWS_SG_IDS[@]}"; do \ aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 15150 --source-group USDAWS_SG_ID --region USDAWS_REGION \ aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 9000 --source-group USDAWS_SG_ID --region USDAWS_REGION \ done The ports are now enabled. 3.3.4. Creating the peer pods secret You must create the peer pods secret for OpenShift sandboxed containers. The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials. Prerequisites You have the following values generated by using the AWS console: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY Procedure Create a peer-pods-secret.yaml manifest file according to the following example: apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2 1 Specify the AWS_ACCESS_KEY_ID value. 2 Specify the AWS_SECRET_ACCESS_KEY value. Create the secret by running the following command: USD oc apply -f peer-pods-secret.yaml 3.3.5. Creating the peer pods config map You must create the peer pods config map for OpenShift sandboxed containers. Prerequisites You have your Amazon Machine Image (AMI) ID if you are not using the default AMI ID based on your cluster credentials. Procedure Obtain the following values from your AWS instance: Retrieve and record the instance ID: USD INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g') This is used to retrieve other values for the secret object. Retrieve and record the AWS region: USD AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"USDAWS_REGION\"" Retrieve and record the AWS subnet ID: USD AWS_SUBNET_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region USD{AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"USDAWS_SUBNET_ID\"" Retrieve and record the AWS VPC ID: USD AWS_VPC_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region USD{AWS_REGION} --output text) && echo "AWS_VPC_ID: \"USDAWS_VPC_ID\"" Retrieve and record the AWS security group IDs: USD AWS_SG_IDS=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region USDAWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"USDAWS_SG_IDS\"" Create a peer-pods-cm.yaml manifest file according to the following example: apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7 DISABLECVM: "true" 1 Defines the default instance type that is used when a type is not defined in the workload. 2 Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads. 3 Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID. 4 Specify the AWS_REGION value you retrieved. 5 Specify the AWS_SUBNET_ID value you retrieved. 6 Specify the AWS_VPC_ID value you retrieved. 7 Specify the AWS_SG_IDS value you retrieved. Create the config map by running the following command: USD oc apply -f peer-pods-cm.yaml 3.3.6. Creating the KataConfig custom resource You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following: Create a RuntimeClass CR named kata-remote with a default configuration. This enables users to configure workloads to use kata-remote as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime. OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime. Important Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows: A larger OpenShift Container Platform deployment with a greater number of worker nodes. Activation of the BIOS and Diagnostics utility. Deployment on a hard disk drive rather than an SSD. Deployment on physical nodes such as bare metal, rather than on virtual nodes. A slow CPU and network. Prerequisites You have access to the cluster as a user with the cluster-admin role. Procedure Create an example-kataconfig.yaml manifest file according to the following example: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info # kataConfigPoolSelector: # matchLabels: # <label_key>: '<label_value>' 1 1 Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true' . Create the KataConfig CR by running the following command: USD oc apply -f example-kataconfig.yaml The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes. Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation. Monitor the installation progress by running the following command: USD watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p" When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster. Verify the daemon set by running the following command: USD oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon Verify the runtime classes by running the following command: USD oc get runtimeclass Example output NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m Verifying the pod VM image After kata-remote is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider. Procedure Obtain the config map you created for the peer pods: USD oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml Check the status stanza of the YAML file. If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully. Troubleshooting Retrieve the events log by running the following command: USD oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation Retrieve the job log by running the following command: USD oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs. 3.3.7. Configuring workload objects You must configure OpenShift sandboxed containers workload objects by setting kata-remote as the runtime class for the following pod-templated objects: Pod objects ReplicaSet objects ReplicationController objects StatefulSet objects Deployment objects DeploymentConfig objects Important Do not deploy workloads in an Operator namespace. Create a dedicated namespace for these resources. You can define whether the workload should be deployed using the default instance type, which you defined in the config map, by adding an annotation to the YAML file. If you do not want to define the instance type manually, you can add an annotation to use an automatic instance type, based on the memory available. Prerequisites You have created the KataConfig custom resource (CR). Procedure Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example: apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ... Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type: To use a manually defined instance type, add the following annotation: apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1 # ... 1 Specify the instance type that you defined in the config map. To use an automatic instance type, add the following annotations: apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ... Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available. Apply the changes to the workload object by running the following command: USD oc apply -f <object.yaml> OpenShift Container Platform creates the workload object and begins scheduling it. Verification Inspect the spec.runtimeClassName field of a pod-templated object. If the value is kata-remote , then the workload is running on OpenShift sandboxed containers, using peer pods.

Commands:
apiVersion: confidentialcontainers.org/v1alpha1 kind: PeerPodConfig metadata: name: peerpodconfig-openshift namespace: openshift-sandboxed-containers-operator spec: cloudSecretName: peer-pods-secret configMapName: peer-pods-cm limit: "10" 1 nodeSelector: node-role.kubernetes.io/kata-oc: ""
INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
AWS_SG_IDS=(USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region USDAWS_REGION))
for AWS_SG_ID in "USD{AWS_SG_IDS[@]}"; do aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 15150 --source-group USDAWS_SG_ID --region USDAWS_REGION aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 9000 --source-group USDAWS_SG_ID --region USDAWS_REGION done
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2
INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"USDAWS_REGION\""
AWS_SUBNET_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region USD{AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"USDAWS_SUBNET_ID\""
AWS_VPC_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region USD{AWS_REGION} --output text) && echo "AWS_VPC_ID: \"USDAWS_VPC_ID\""
AWS_SG_IDS=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region USDAWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"USDAWS_SG_IDS\""
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7 DISABLECVM: "true"
oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
apiVersion: v1 kind: <object> spec: runtimeClassName: kata-remote
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory>
apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
oc apply -f osc-namespace.yaml
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: sandboxed-containers-operator-group namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
oc apply -f osc-operatorgroup.yaml
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.8.1
oc apply -f osc-subscription.yaml
oc get csv -n openshift-sandboxed-containers-operator
watch oc get csv -n openshift-sandboxed-containers-operator
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.8.1 1.7.0 Succeeded
oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator -o jsonpath='{.spec.limit}{"\n"}'
oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator --type merge --patch '{"spec":{"limit":"<value>"}}' 1
INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
AWS_SG_IDS=(USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region USDAWS_REGION))
for AWS_SG_ID in "USD{AWS_SG_IDS[@]}"; do aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 15150 --source-group USDAWS_SG_ID --region USDAWS_REGION aws ec2 authorize-security-group-ingress --group-id USDAWS_SG_ID --protocol tcp --port 9000 --source-group USDAWS_SG_ID --region USDAWS_REGION done
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2
oc apply -f peer-pods-secret.yaml
INSTANCE_ID=USD(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
AWS_REGION=USD(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"USDAWS_REGION\""
AWS_SUBNET_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region USD{AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"USDAWS_SUBNET_ID\""
AWS_VPC_ID=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region USD{AWS_REGION} --output text) && echo "AWS_VPC_ID: \"USDAWS_VPC_ID\""
AWS_SG_IDS=USD(aws ec2 describe-instances --instance-ids USD{INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region USDAWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"USDAWS_SG_IDS\""
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7 DISABLECVM: "true"
oc apply -f peer-pods-cm.yaml
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
oc apply -f example-kataconfig.yaml
watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon
oc get runtimeclass
NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m
oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml
oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
apiVersion: v1 kind: <object> spec: runtimeClassName: kata-remote
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory>
oc apply -f <object.yaml>