Chapter 7. Deploying Confidential Containers on IBM Z and IBM LinuxONE You can deploy Confidential Containers on IBM Z(R) and IBM(R) LinuxONE after you deploy OpenShift sandboxed containers. Important Confidential Containers on IBM Z(R) and IBM(R) LinuxONE is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Cluster requirements You have installed Red Hat OpenShift Container Platform 4.15 or later on the cluster where you are installing the Confidential compute attestation Operator. You deploy Confidential Containers by performing the following steps: Install the Confidential compute attestation Operator. Create the route for Trustee. Enable the Confidential Containers feature gate. Update the peer pods config map. Delete the KataConfig custom resource (CR). Update the peer pods secret. Re-create the KataConfig CR. Create the Trustee authentication secret. Create the Trustee config map. Obtain the IBM Secure Execution (SE) header. Configure the SE certificates and keys. Configure Trustee values, policies, and secrets. Create the KbsConfig CR. Verify the Trustee configuration. Verify the attestation process. 7.1. Installing the Confidential compute attestation Operator You can install the Confidential compute attestation Operator on IBM Z(R) and IBM(R) LinuxONE by using the CLI. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Create a trustee-namespace.yaml manifest file: apiVersion: v1 kind: Namespace metadata: name: trustee-operator-system Create the trustee-operator-system namespace by running the following command: USD oc apply -f trustee-namespace.yaml Create a trustee-operatorgroup.yaml manifest file: apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: trustee-operator-group namespace: trustee-operator-system spec: targetNamespaces: - trustee-operator-system Create the operator group by running the following command: USD oc apply -f trustee-operatorgroup.yaml Create a trustee-subscription.yaml manifest file: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: trustee-operator namespace: trustee-operator-system spec: channel: stable installPlanApproval: Automatic name: trustee-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: trustee-operator.v0.1.0 Create the subscription by running the following command: USD oc apply -f trustee-subscription.yaml Verify that the Operator is correctly installed by running the following command: USD oc get csv -n trustee-operator-system This command can take several minutes to complete. Watch the process by running the following command: USD watch oc get csv -n trustee-operator-system Example output NAME DISPLAY PHASE trustee-operator.v0.1.0 Trustee Operator 0.1.0 Succeeded 7.2. Enabling the Confidential Containers feature gate You must enable the Confidential Containers feature gate. Prerequisites You have subscribed to the OpenShift sandboxed containers Operator. Procedure Create a cc-feature-gate.yaml manifest file: apiVersion: v1 kind: ConfigMap metadata: name: osc-feature-gates namespace: openshift-sandboxed-containers-operator data: confidential: "true" Create the config map by running the following command: USD oc apply -f cc-feature-gate.yaml 7.3. Creating the route for Trustee You can create a secure route with edge TLS termination for Trustee. External ingress traffic reaches the router pods as HTTPS and passes on to the Trustee pods as HTTP. Prerequisites You have installed the Confidential compute attestation Operator. Procedure Create an edge route by running the following command: USD oc create route edge --service=kbs-service --port kbs-port \ -n trustee-operator-system Note Note: Currently, only a route with a valid CA-signed certificate is supported. You cannot use a route with self-signed certificate. Set the TRUSTEE_HOST variable by running the following command: USD TRUSTEE_HOST=USD(oc get route -n trustee-operator-system kbs-service \ -o jsonpath={.spec.host}) Verify the route by running the following command: USD echo USDTRUSTEE_HOST Example output kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io 7.4. Updating the peer pods config map You must update the peer pods config map for Confidential Containers. Note Set Secure Boot to true to enable it by default. The default value is false , which presents a security risk. Procedure Create a peer-pods-cm.yaml manifest file according to the following example: apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "libvirt" DISABLECVM: "false" AA_KBC_PARAMS: "cc_kbc::https://USD{TRUSTEE_HOST}" 1 1 Specify the host name of the Trustee route. Create the config map by running the following command: USD oc apply -f peer-pods-cm.yaml Restart the peerpodconfig-ctrl-caa-daemon daemon set by running the following command: USD oc set env ds/peerpodconfig-ctrl-caa-daemon \ -n openshift-sandboxed-containers-operator REBOOT="USD(date)" 7.5. Deleting the KataConfig custom resource You can delete the KataConfig custom resource (CR) by using the command line. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Delete the KataConfig CR by running the following command: USD oc delete kataconfig example-kataconfig Verify that the custom resource was deleted by running the following command: USD oc get kataconfig example-kataconfig Example output No example-kataconfig instances exist 7.6. Updating the peer pods secret You must update the peer pods secret for Confidential Containers. The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials. Prerequisites REDHAT_OFFLINE_TOKEN . You have generated this token to download the RHEL image at Red Hat API Tokens . HKD_CRT . The Host Key Document (HKD) certificate enables secure execution on IBM Z(R). For more information, see Obtaining a host key document from Resource Link in the IBM documentation. Procedure Create a peer-pods-secret.yaml manifest file according to the following example: apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 1 HKD_CRT: "<hkd_crt_value>" 2 1 Specify the Red Hat offline token, which is required for the Operator-built image. 2 Specify the HKD certificate value to enable IBM Secure Execution for the Operator-built image. Create the secret by running the following command: USD oc apply -f peer-pods-secret.yaml 7.7. Re-creating the KataConfig custom resource You must re-create the KataConfig custom resource (CR) for Confidential Containers. Important Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows: A larger OpenShift Container Platform deployment with a greater number of worker nodes. Activation of the BIOS and Diagnostics utility. Deployment on a hard disk drive rather than an SSD. Deployment on physical nodes such as bare metal, rather than on virtual nodes. A slow CPU and network. Prerequisites You have access to the cluster as a user with the cluster-admin role. Procedure Create an example-kataconfig.yaml manifest file according to the following example: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info # kataConfigPoolSelector: # matchLabels: # <label_key>: '<label_value>' 1 1 Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true' . Create the KataConfig CR by running the following command: USD oc apply -f example-kataconfig.yaml The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes. Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation. Monitor the installation progress by running the following command: USD watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p" When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command: USD oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator Example output Name: peer-pods-cm Namespace: openshift-sandboxed-containers-operator Labels: <none> Annotations: <none> Data ==== CLOUD_PROVIDER: libvirt DISABLECVM: false 1 LIBVIRT_IMAGE_ID: fa-pp-vol 2 BinaryData ==== Events: <none> 1 Enables the Confidential VM during IBM Secure Execution for the Operator-built image. 2 Contains a value if you have built the peer pod image and uploaded it to the libvirt volume. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT , by running the following command: USD watch oc get mcp/kata-oc Verify the daemon set by running the following command: USD oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon Verify the runtime classes by running the following command: USD oc get runtimeclass Example output NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m 7.8. Creating the Trustee authentication secret You must create the authentication secret for Trustee. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Create a private key by running the following command: USD openssl genpkey -algorithm ed25519 > privateKey Create a public key by running the following command: USD openssl pkey -in privateKey -pubout -out publicKey Create a secret by running the following command: USD oc create secret generic kbs-auth-public-key --from-file=publicKey -n trustee-operator-system Verify the secret by running the following command: USD oc get secret -n trustee-operator-system 7.9. Creating the Trustee config map You must create the config map to configure the Trustee server. Note The following configuration example turns off security features to enable demonstration of Technology Preview features. It is not meant for a production environment. Prerequisites You have created a route for Trustee. Procedure Create a kbs-config-cm.yaml manifest file: apiVersion: v1 kind: ConfigMap metadata: name: kbs-config-cm namespace: trustee-operator-system data: kbs-config.json: | { "insecure_http" : true, "sockets": ["0.0.0.0:8080"], "auth_public_key": "/etc/auth-secret/publicKey", "attestation_token_config": { "attestation_token_type": "CoCo" }, "repository_config": { "type": "LocalFs", "dir_path": "/opt/confidential-containers/kbs/repository" }, "as_config": { "work_dir": "/opt/confidential-containers/attestation-service", "policy_engine": "opa", "attestation_token_broker": "Simple", "attestation_token_config": { "duration_min": 5 }, "rvps_config": { "store_type": "LocalJson", "store_config": { "file_path": "/opt/confidential-containers/rvps/reference-values/reference-values.json" } } }, "policy_engine_config": { "policy_path": "/opt/confidential-containers/opa/policy.rego" } } Create the config map by running the following command: USD oc apply -f kbs-config-cm.yaml 7.10. Configuring the IBM Secure Execution certificates and keys You must configure the IBM Secure Execution (SE) certificates and keys for your worker nodes. Prerequisites You have the IP address of the bastion node. You have the internal IP addresses of the worker nodes. Procedure Obtain the attestation policy fields by performing the following steps: Create a directory to download the GetRvps.sh script by running the following command: USD mkdir -p Rvps-Extraction/ Download the script by running the following command: USD wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/GetRvps.sh -O USDPWD/GetRvps.sh Create a subdirectory by running the following command: USD mkdir -p Rvps-Extraction/static-files Go to the static-files directory by running the following command: USD cd Rvps-Extraction/static-files Download the pvextract-hdr tool by running the following command: USD wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/pvextract-hdr -O USDPWD/pvextract-hdr Make the tool executable by running the following command: USD chmod +x pvextract-hdr Download the se_parse_hdr.py script by running the following command: USD wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/se_parse_hdr.py -O USDPWD/se_parse_hdr.py Copy your Host Key Document (HKD) certificate to the static-files directory by running the following command: USD cp ~/path/to/<hkd_cert.crt> . The static-files directory contains the following files: HKD.crt pvextract-hdr se_parse_hdr.py Go to the Rvps-Extraction directory by running the following command: USD cd .. Make the GetRvps.sh script executable by running the following command: USD chmod +x GetRvps.sh Run the script: USD ./GetRvps.sh Example output ***Installing necessary packages for RVPS values extraction *** Updating Subscription Management repositories. Last metadata expiration check: 0:37:12 ago on Mon Nov 18 09:20:29 2024. Package python3-3.9.19-8.el9_5.1.s390x is already installed. Package python3-cryptography-36.0.1-4.el9.s390x is already installed. Package kmod-28-10.el9.s390x is already installed. Dependencies resolved. Nothing to do. Complete! ***Installation Finished *** 1) Generate the RVPS From Local Image from User pc 2) Generate RVPS from Volume 3) Quit Please enter your choice: Enter 2 to generate the Reference Value Provider Service from the volume: Please enter your choice: 2 Enter fa-pp for the libvirt pool name: Enter the Libvirt Pool Name: fa-pp Enter the libvirt gateway URI: Enter the Libvirt URI Name: <libvirt-uri> 1 1 Specify the LIBVIRT_URI value that you used to create the peer pods secret . Enter fa-pp-vol for the libvirt volume name: Enter the Libvirt Volume Name: fa-pp-vol Example output Downloading from PODVM Volume... mount: /mnt/myvm: special device /dev/nbd3p1 does not exist. Error: Failed to mount the image. Retrying... Mounting on second attempt passed /dev/nbd3 disconnected SE header found at offset 0x014000 SE header written to '/root/Rvps-Extraction/output-files/hdr.bin' (640 bytes) se.tag: 42f3fe61e8a7e859cab3bb033fd11c61 se.image_phkh: 92d0aff6eb86719b6b1ea0cb98d2c99ff2ec693df3efff2158f54112f6961508 provenance = ewogICAgInNlLmF0dGVzdGF0aW9uX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnRhZyI6IFsKICAgICAgICAiNDJmM2ZlNjFlOGE3ZTg1OWNhYjNiYjAzM2ZkMTFjNjEiCiAgICBdLAogICAgInNlLmltYWdlX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnVzZXJfZGF0YSI6IFsKICAgICAgICAiMDAiCiAgICBdLAogICAgInNlLnZlcnNpb24iOiBbCiAgICAgICAgIjI1NiIKICAgIF0KfQo= -rw-r--r--. 1 root root 640 Dec 16 10:57 /root/Rvps-Extraction/output-files/hdr.bin -rw-r--r--. 1 root root 446 Dec 16 10:57 /root/Rvps-Extraction/output-files/ibmse-policy.rego -rw-r--r--. 1 root root 561 Dec 16 10:57 /root/Rvps-Extraction/output-files/se-message Obtain the certificates and certificate revocation lists (CRLs) by performing the following steps: Create a temporary directory for certificates by running the following command: USD mkdir /tmp/ibmse/certs Download the ibm-z-host-key-signing-gen2.crt certificate by running the following command: USD wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt Download the DigiCertCA.crt certificate by running the following command: USD wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt Create a temporary directory for the CRLs by running the following command: USD mkdir /tmp/ibmse/crls Download the ibm-z-host-key-gen2.crl file by running the following command: USD wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-gen2.crl -O /tmp/ibmse/crls/ibm-z-host-key-gen2.crl Download the DigiCertTrustedRootG4.crl file by running the following command: USD wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl Download the DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl file by running the following command: USD wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl Create a temporary directory for the hdr.bin file by running the following command: USD mkdir -p /tmp/ibmse/hdr/ Copy the hdr.bin file to the hdr directory by running the following command: USD cp /root/Rvps-Extraction/output-files/hdr.bin /tmp/ibmse/hdr/ Create a temporary directory for Host Key Document (HKD) certificate by running the following command: USD mkdir -p /tmp/ibmse/hkds Copy your HKD certificate to the hkds directory by running the following command: USD cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/ Generate the RSA keys: Generate an RSA key pair by running the following command: USD openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 1 1 Specify the RSA key password. Create a temporary directory for the RSA keys by running the following command: USD mkdir -p /tmp/ibmse/rsa Create an encrypt_key.pub key by running the following command: USD openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub Create an encrypt_key.pem key by running the following command: USD openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem Verify the structure of the /tmp/ibmse directory by running the following command: USD tree /tmp/ibmse Example output /tmp/ibmse ├── certs │ ├── ibm-z-host-key-signing-gen2.crt | └── DigiCertCA.crt ├── crls │ └── ibm-z-host-key-gen2.crl │ └── DigiCertTrustedRootG4.crl │ └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl ├── hdr │ └── hdr.bin ├── hkds │ └── <hkd_cert.crt> └── rsa ├── encrypt_key.pem └── encrypt_key.pub Copy these files to the OpenShift Container Platform worker nodes by performing the following steps: Create a compressed file from the /tmp/ibmse directory by running the following command: USD tar -czf ibmse.tar.gz -C /tmp/ ibmse Copy the .tar.gz file to the bastion node in your cluster by running the following command: USD scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 1 1 Specify the IP address of the bastion node. Connect to the bastion node over SSH by running the following command: USD ssh root@<ocp_bastion_ip> Copy the .tar.gz file to each worker node by running the following command: USD scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 1 1 Specify the IP address of the worker node. Extract the .tar.gz on each worker node by running the following command: USD ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/' Update the ibmse folder permissions by running the following command: USD ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/' 7.11. Creating the persistent storage components You must create persistent storage components, persistent volume (PV) and persistent volume claim (PVC) to mount the ibmse folder to the Trustee pod. Procedure Create a persistent-volume.yaml manifest file: apiVersion: v1 kind: PersistentVolume metadata: name: ibmse-pv namespace: trustee-operator-system spec: capacity: storage: 100Mi accessModes: - ReadOnlyMany storageClassName: "" local: path: /opt/confidential-containers/ibmse nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists Create the persistent volume by running the following command: USD oc apply -f persistent-volume.yaml Create a persistent-volume-claim.yaml manifest file: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ibmse-pvc namespace: trustee-operator-system spec: accessModes: - ReadOnlyMany storageClassName: "" resources: requests: storage: 100Mi Create the persistent volume claim by running the following command: USD oc apply -f persistent-volume-claim.yaml 7.12. Configuring Trustee values, policies, and secrets You can configure the following values, policies, and secrets for Trustee: Optional: Reference values for the Reference Value Provider Service. Attestation policy for IBM Secure Execution. Optional: Secret for custom keys for Trustee clients. Optional: Secret for container image signature verification. Container image signature verification policy. This policy is mandatory. If you do not use container image signature verification, you must create a policy that does not verify signatures. Resource access policy. 7.12.1. Configuring reference values You can configure reference values for the Reference Value Provider Service (RVPS) by specifying the trusted digests of your hardware platform. The client collects measurements from the running software, the Trusted Execution Environment (TEE) hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to the Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with. Procedure Create an rvps-configmap.yaml manifest file: apiVersion: v1 kind: ConfigMap metadata: name: rvps-reference-values namespace: trustee-operator-system data: reference-values.json: | [ 1 ] 1 Leave this value empty. Create the RVPS config map by running the following command: USD oc apply -f rvps-configmap.yaml 7.12.2. Creating the attestation policy for IBM Secure Execution You must create the attestation policy for IBM Secure Execution. Procedure Create an attestation-policy.yaml manifest file: apiVersion: v1 kind: ConfigMap metadata: name: attestation-policy namespace: trustee-operator-system data: default.rego: | 1 package policy import rego.v1 default allow = false converted_version := sprintf("%v", [input["se.version"]]) allow if { input["se.attestation_phkh"] == "<se.attestation_phkh>" 2 input["se.image_phkh"] == "<se.image_phkh>" input["se.tag"] == "<se.tag>" converted_version == "256" } 1 Do not modify the policy name. 2 Specify the attestation policy fields you obtained by running the se_parse_hdr.py script. Create the attestation policy config map by running the following command: USD oc apply -f attestation-policy.yaml 7.12.3. Creating a secret with custom keys for clients You can create a secret that contains one or more custom keys for Trustee clients. In this example, the kbsres1 secret has two entries ( key1 , key2 ), which the clients retrieve. You can add additional secrets according to your requirements by using the same format. Prerequisites You have created one or more custom keys. Procedure Create a secret for the custom keys according to the following example: USD oc apply secret generic kbsres1 \ --from-literal key1=<custom_key1> \ 1 --from-literal key2=<custom_key2> \ -n trustee-operator-system 1 Specify a custom key. The kbsres1 secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource. 7.12.4. Creating a secret for container image signature verification If you use container image signature verification, you must create a secret that contains the public container image signing key. The Confidential compute attestation Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment. You can use Red Hat Trusted Artifact Signer or other tools to sign container images. Procedure Create a secret for container image signature verification by running the following command: USD oc apply secret generic <type> \ 1 --from-file=<tag>=./<public_key_file> \ 2 -n trustee-operator-system 1 Specify the KBS secret type, for example, img-sig . 2 Specify the secret tag, for example, pub-key , and the public container image signing key. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource. 7.12.5. Creating the container image signature verification policy You create the container image signature verification policy because signature verification is always enabled. If this policy is missing, the pods will not start. If you are not using container image signature verification, you create the policy without signature verification. For more information, see containers-policy.json 5 . Procedure Create a security-policy-config.json file according to the following examples: Without signature verification: { "default": [ { "type": "insecureAcceptAnything" }], "transports": {} } With signature verification: { "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "<transport>": { 1 "<registry>/<image>": 2 [ { "type": "sigstoreSigned", "keyPath": "kbs:///default/<type>/<tag>" 3 } ] } } } 1 Specify the image repository for transport , for example, "docker": . For more information, see containers-transports 5 . 2 Specify the container registry and image, for example, "quay.io/my-image". 3 Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key . Create the security policy by running the following command: USD oc apply secret generic security-policy \ --from-file=osc=./<security-policy-config.json> \ -n trustee-operator-system Do not alter the secret type, security-policy , or the key, osc . The security-policy secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource. 7.12.6. Creating the resource access policy You configure the resource access policy for the Trustee policy engine. This policy determines which resources Trustee can access. Note The Trustee policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence. Procedure Create a resourcepolicy-configmap.yaml manifest file: apiVersion: v1 kind: ConfigMap metadata: name: resource-policy namespace: trustee-operator-system data: policy.rego: | 1 package policy 2 path := split(data["resource-path"], "/") default allow = false allow { count(path) == 3 input["tee"] == "se" } 1 The name of the resource policy, policy.rego , must match the resource policy defined in the Trustee config map. 2 The resource policy follows the Open Policy Agent specification. This example allows the retrieval of all resources when the TEE is not the sample attester. Create the resource policy config map by running the following command: USD oc apply -f resourcepolicy-configmap.yaml 7.13. Creating the KbsConfig custom resource You create the KbsConfig custom resource (CR) to launch Trustee. Then, you check the Trustee pods and pod logs to verify the configuration. Procedure Create a kbsconfig-cr.yaml manifest file: apiVersion: confidentialcontainers.org/v1alpha1 kind: KbsConfig metadata: labels: app.kubernetes.io/name: kbsconfig app.kubernetes.io/instance: kbsconfig app.kubernetes.io/part-of: trustee-operator app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: trustee-operator name: kbsconfig namespace: trustee-operator-system spec: kbsConfigMapName: kbs-config-cm kbsAuthSecretName: kbs-auth-public-key kbsDeploymentType: AllInOneDeployment kbsRvpsRefValuesConfigMapName: rvps-reference-values kbsSecretResources: ["kbsres1", "security-policy", "<type>"] 1 kbsResourcePolicyConfigMapName: resource-policy kbsAttestationPolicyConfigMapName: attestation-policy kbsServiceType: NodePort ibmSEConfigSpec: certStorePvc: ibmse-pvc KbsEnvVars: SE_SKIP_CERTS_VERIFICATION: "true" 1 Optional: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig . If you did not create the secret, set the kbsSecretResources value to ["kbsres1", "security-policy"] . Create the KbsConfig CR by running the following command: USD oc apply -f kbsconfig-cr.yaml 7.14. Verifying the Trustee configuration You verify the Trustee configuration by checking the Trustee pods and logs. Procedure Set the default project by running the following command: USD oc project trustee-operator-system Check the Trustee pods by running the following command: USD oc get pods -n trustee-operator-system Example output NAME READY STATUS RESTARTS AGE trustee-deployment-8585f98449-9bbgl 1/1 Running 0 22m trustee-operator-controller-manager-5fbd44cd97-55dlh 2/2 Running 0 59m Set the POD_NAME environmental variable by running the following command: USD POD_NAME=USD(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system) Check the pod logs by running the following command: USD oc logs -n trustee-operator-system USDPOD_NAME Example output [2024-05-30T13:44:24Z INFO kbs] Using config file /etc/kbs-config/kbs-config.json [2024-05-30T13:44:24Z WARN attestation_service::rvps] No RVPS address provided and will launch a built-in rvps [2024-05-30T13:44:24Z INFO attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert [2024-05-30T13:44:24Z INFO api_server] Starting HTTPS server at [0.0.0.0:8080] [2024-05-30T13:44:24Z INFO actix_server::builder] starting 12 workers [2024-05-30T13:44:24Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime Verify that the kbs-service is exposed on a node port by running the following command: USD oc get svc kbs-service -n trustee-operator-system Example output NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kbs-service NodePort 198.51.100.54 <none> 8080:31862/TCP 23h Obtain the Trustee deployment pod name by running the following command: USD oc get pods -n trustee-operator-system | grep -i trustee-deployment Example output NAME READY STATUS RESTARTS AGE trustee-deployment-d746679cd-plq82 1/1 Running 0 2m32s Obtain the worker node IP address by running the following command: USD oc get pod trustee-deployment-d746679cd-plq82 -o custom-columns="NODE-IP:.status.hostIP" Example output NODE-IP 192.168.122.36 The URL to access the Trustee is http://<worker_node_ip>:<node_port> , for example, http://192.168.122.36:31862 . Verify that the AA_KBC_PARAMS value in the peer-pods-cm config map is the same as the Trustee URL by running the following command: USD oc get cm peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml | grep AA_KBC_PARAMS Example output AA_KBC_PARAMS: cc_kbc::http://192.168.122.36:31862 7.15. Verifying the attestation process You can verify the attestation process by creating a BusyBox pod. The pod image deploys the confidential workload where you can retrieve the key. Important This procedure is an example to verify that attestation is working. Do not write sensitive data to standard I/O because the data can be captured by using a memory dump. Only data written to memory is encrypted. Procedure Create a busybox.yaml manifest file: apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: runtimeClassName: kata-remote containers: - name: busybox image: quay.io/prometheus/busybox:latest imagePullPolicy: Always command: - "sleep" - "3600" Create the pod by running the following command: USD oc create -f busybox.yaml Log in to the pod by running the following command: USD oc exec -it busybox -n default -- /bin/sh Get the secret key by running the following command: USD wget http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1 Example output Connecting to 127.0.0.1:8006 (127.0.0.1:8006) saving to 'key1' key1 100% |*******************************************| 8 0:00:00 ETA 'key1' saved Display the key1 value by running the following command: USD cat key1 Example output res1val1/ #

Commands:
apiVersion: v1 kind: Namespace metadata: name: trustee-operator-system
oc apply -f trustee-namespace.yaml
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: trustee-operator-group namespace: trustee-operator-system spec: targetNamespaces: - trustee-operator-system
oc apply -f trustee-operatorgroup.yaml
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: trustee-operator namespace: trustee-operator-system spec: channel: stable installPlanApproval: Automatic name: trustee-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: trustee-operator.v0.1.0
oc apply -f trustee-subscription.yaml
oc get csv -n trustee-operator-system
watch oc get csv -n trustee-operator-system
NAME DISPLAY PHASE trustee-operator.v0.1.0 Trustee Operator 0.1.0 Succeeded
apiVersion: v1 kind: ConfigMap metadata: name: osc-feature-gates namespace: openshift-sandboxed-containers-operator data: confidential: "true"
oc apply -f cc-feature-gate.yaml
oc create route edge --service=kbs-service --port kbs-port -n trustee-operator-system
TRUSTEE_HOST=USD(oc get route -n trustee-operator-system kbs-service -o jsonpath={.spec.host})
echo USDTRUSTEE_HOST
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "libvirt" DISABLECVM: "false" AA_KBC_PARAMS: "cc_kbc::https://USD{TRUSTEE_HOST}" 1
oc apply -f peer-pods-cm.yaml
oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="USD(date)"
oc delete kataconfig example-kataconfig
oc get kataconfig example-kataconfig
No example-kataconfig instances exist
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 1 HKD_CRT: "<hkd_crt_value>" 2
oc apply -f peer-pods-secret.yaml
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
oc apply -f example-kataconfig.yaml
watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator
Name: peer-pods-cm Namespace: openshift-sandboxed-containers-operator Labels: <none> Annotations: <none> Data ==== CLOUD_PROVIDER: libvirt DISABLECVM: false 1 LIBVIRT_IMAGE_ID: fa-pp-vol 2 BinaryData ==== Events: <none>
watch oc get mcp/kata-oc
oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon
oc get runtimeclass
NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m
openssl genpkey -algorithm ed25519 > privateKey
openssl pkey -in privateKey -pubout -out publicKey
oc create secret generic kbs-auth-public-key --from-file=publicKey -n trustee-operator-system
oc get secret -n trustee-operator-system
apiVersion: v1 kind: ConfigMap metadata: name: kbs-config-cm namespace: trustee-operator-system data: kbs-config.json: | { "insecure_http" : true, "sockets": ["0.0.0.0:8080"], "auth_public_key": "/etc/auth-secret/publicKey", "attestation_token_config": { "attestation_token_type": "CoCo" }, "repository_config": { "type": "LocalFs", "dir_path": "/opt/confidential-containers/kbs/repository" }, "as_config": { "work_dir": "/opt/confidential-containers/attestation-service", "policy_engine": "opa", "attestation_token_broker": "Simple", "attestation_token_config": { "duration_min": 5 }, "rvps_config": { "store_type": "LocalJson", "store_config": { "file_path": "/opt/confidential-containers/rvps/reference-values/reference-values.json" } } }, "policy_engine_config": { "policy_path": "/opt/confidential-containers/opa/policy.rego" } }
oc apply -f kbs-config-cm.yaml
mkdir -p Rvps-Extraction/
wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/GetRvps.sh -O USDPWD/GetRvps.sh
mkdir -p Rvps-Extraction/static-files
cd Rvps-Extraction/static-files
wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/pvextract-hdr -O USDPWD/pvextract-hdr
chmod +x pvextract-hdr
wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/se_parse_hdr.py -O USDPWD/se_parse_hdr.py
cp ~/path/to/<hkd_cert.crt> .
cd ..
chmod +x GetRvps.sh
./GetRvps.sh
***Installing necessary packages for RVPS values extraction *** Updating Subscription Management repositories. Last metadata expiration check: 0:37:12 ago on Mon Nov 18 09:20:29 2024. Package python3-3.9.19-8.el9_5.1.s390x is already installed. Package python3-cryptography-36.0.1-4.el9.s390x is already installed. Package kmod-28-10.el9.s390x is already installed. Dependencies resolved. Nothing to do. Complete! ***Installation Finished *** 1) Generate the RVPS From Local Image from User pc 2) Generate RVPS from Volume 3) Quit Please enter your choice:
Please enter your choice: 2
Enter the Libvirt Pool Name: fa-pp
Enter the Libvirt URI Name: <libvirt-uri> 1
Enter the Libvirt Volume Name: fa-pp-vol
Downloading from PODVM Volume mount: /mnt/myvm: special device /dev/nbd3p1 does not exist. Error: Failed to mount the image. Retrying Mounting on second attempt passed /dev/nbd3 disconnected SE header found at offset 0x014000 SE header written to '/root/Rvps-Extraction/output-files/hdr.bin' (640 bytes) se.tag: 42f3fe61e8a7e859cab3bb033fd11c61 se.image_phkh: 92d0aff6eb86719b6b1ea0cb98d2c99ff2ec693df3efff2158f54112f6961508 provenance = ewogICAgInNlLmF0dGVzdGF0aW9uX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnRhZyI6IFsKICAgICAgICAiNDJmM2ZlNjFlOGE3ZTg1OWNhYjNiYjAzM2ZkMTFjNjEiCiAgICBdLAogICAgInNlLmltYWdlX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnVzZXJfZGF0YSI6IFsKICAgICAgICAiMDAiCiAgICBdLAogICAgInNlLnZlcnNpb24iOiBbCiAgICAgICAgIjI1NiIKICAgIF0KfQo= -rw-r--r--. 1 root root 640 Dec 16 10:57 /root/Rvps-Extraction/output-files/hdr.bin -rw-r--r--. 1 root root 446 Dec 16 10:57 /root/Rvps-Extraction/output-files/ibmse-policy.rego -rw-r--r--. 1 root root 561 Dec 16 10:57 /root/Rvps-Extraction/output-files/se-message
mkdir /tmp/ibmse/certs
wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt
wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt
mkdir /tmp/ibmse/crls
wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-gen2.crl -O /tmp/ibmse/crls/ibm-z-host-key-gen2.crl
wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl
wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
mkdir -p /tmp/ibmse/hdr/
cp /root/Rvps-Extraction/output-files/hdr.bin /tmp/ibmse/hdr/
mkdir -p /tmp/ibmse/hkds
cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/
openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 1
mkdir -p /tmp/ibmse/rsa
openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub
openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem
tree /tmp/ibmse
/tmp/ibmse ├── certs │ ├── ibm-z-host-key-signing-gen2.crt | └── DigiCertCA.crt ├── crls │ └── ibm-z-host-key-gen2.crl │ └── DigiCertTrustedRootG4.crl │ └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl ├── hdr │ └── hdr.bin ├── hkds │ └── <hkd_cert.crt> └── rsa ├── encrypt_key.pem └── encrypt_key.pub
tar -czf ibmse.tar.gz -C /tmp/ ibmse
scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 1
ssh root@<ocp_bastion_ip>
scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 1
ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/'
ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/'
apiVersion: v1 kind: PersistentVolume metadata: name: ibmse-pv namespace: trustee-operator-system spec: capacity: storage: 100Mi accessModes: - ReadOnlyMany storageClassName: "" local: path: /opt/confidential-containers/ibmse nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists
oc apply -f persistent-volume.yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ibmse-pvc namespace: trustee-operator-system spec: accessModes: - ReadOnlyMany storageClassName: "" resources: requests: storage: 100Mi
oc apply -f persistent-volume-claim.yaml
apiVersion: v1 kind: ConfigMap metadata: name: rvps-reference-values namespace: trustee-operator-system data: reference-values.json: | [ 1 ]
oc apply -f rvps-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: attestation-policy namespace: trustee-operator-system data: default.rego: | 1 package policy import rego.v1 default allow = false converted_version := sprintf("%v", [input["se.version"]]) allow if { input["se.attestation_phkh"] == "<se.attestation_phkh>" 2 input["se.image_phkh"] == "<se.image_phkh>" input["se.tag"] == "<se.tag>" converted_version == "256" }
oc apply -f attestation-policy.yaml
oc apply secret generic kbsres1 --from-literal key1=<custom_key1> \ 1 --from-literal key2=<custom_key2> -n trustee-operator-system
oc apply secret generic <type> \ 1 --from-file=<tag>=./<public_key_file> \ 2 -n trustee-operator-system
{ "default": [ { "type": "insecureAcceptAnything" }], "transports": {} }
{ "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "<transport>": { 1 "<registry>/<image>": 2 [ { "type": "sigstoreSigned", "keyPath": "kbs:///default/<type>/<tag>" 3 } ] } } }
oc apply secret generic security-policy --from-file=osc=./<security-policy-config.json> -n trustee-operator-system
apiVersion: v1 kind: ConfigMap metadata: name: resource-policy namespace: trustee-operator-system data: policy.rego: | 1 package policy 2 path := split(data["resource-path"], "/") default allow = false allow { count(path) == 3 input["tee"] == "se" }
oc apply -f resourcepolicy-configmap.yaml
apiVersion: confidentialcontainers.org/v1alpha1 kind: KbsConfig metadata: labels: app.kubernetes.io/name: kbsconfig app.kubernetes.io/instance: kbsconfig app.kubernetes.io/part-of: trustee-operator app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: trustee-operator name: kbsconfig namespace: trustee-operator-system spec: kbsConfigMapName: kbs-config-cm kbsAuthSecretName: kbs-auth-public-key kbsDeploymentType: AllInOneDeployment kbsRvpsRefValuesConfigMapName: rvps-reference-values kbsSecretResources: ["kbsres1", "security-policy", "<type>"] 1 kbsResourcePolicyConfigMapName: resource-policy kbsAttestationPolicyConfigMapName: attestation-policy kbsServiceType: NodePort ibmSEConfigSpec: certStorePvc: ibmse-pvc KbsEnvVars: SE_SKIP_CERTS_VERIFICATION: "true"
oc apply -f kbsconfig-cr.yaml
oc project trustee-operator-system
oc get pods -n trustee-operator-system
NAME READY STATUS RESTARTS AGE trustee-deployment-8585f98449-9bbgl 1/1 Running 0 22m trustee-operator-controller-manager-5fbd44cd97-55dlh 2/2 Running 0 59m
POD_NAME=USD(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
oc logs -n trustee-operator-system USDPOD_NAME
[2024-05-30T13:44:24Z INFO kbs] Using config file /etc/kbs-config/kbs-config.json [2024-05-30T13:44:24Z WARN attestation_service::rvps] No RVPS address provided and will launch a built-in rvps [2024-05-30T13:44:24Z INFO attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert [2024-05-30T13:44:24Z INFO api_server] Starting HTTPS server at [0.0.0.0:8080] [2024-05-30T13:44:24Z INFO actix_server::builder] starting 12 workers [2024-05-30T13:44:24Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime
oc get svc kbs-service -n trustee-operator-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kbs-service NodePort 198.51.100.54 <none> 8080:31862/TCP 23h
oc get pods -n trustee-operator-system | grep -i trustee-deployment
NAME READY STATUS RESTARTS AGE trustee-deployment-d746679cd-plq82 1/1 Running 0 2m32s
oc get pod trustee-deployment-d746679cd-plq82 -o custom-columns="NODE-IP:.status.hostIP"
NODE-IP 192.168.122.36
oc get cm peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml | grep AA_KBC_PARAMS
AA_KBC_PARAMS: cc_kbc::http://192.168.122.36:31862
apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: runtimeClassName: kata-remote containers: - name: busybox image: quay.io/prometheus/busybox:latest imagePullPolicy: Always command: - "sleep" - "3600"
oc create -f busybox.yaml
oc exec -it busybox -n default -- /bin/sh
wget http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1
Connecting to 127.0.0.1:8006 (127.0.0.1:8006) saving to 'key1' key1 100% |*******************************************| 8 0:00:00 ETA 'key1' saved
cat key1
res1val1/ #