Chapter 8. NBDE Tang Server Operator 8.1. NBDE Tang Server Operator overview Network-bound Disk Encryption (NBDE) provides an automated unlocking of LUKS-encrypted volumes using one or more dedicated network-binding servers. The client side of NBDE is called the Clevis decryption policy framework and the server side is represented by Tang. The NBDE Tang Server Operator allows the automation of deployments of one or several Tang servers in the OpenShift Container Platform (OCP) environment. 8.2. NBDE Tang Server Operator release notes The following release notes track the development of the NBDE Tang Server Operator in OpenShift Container Platform. RHEA-2023:7491 The NBDE Tang Server Operator 1.0 has been released in the Red Hat OpenShift Enterprise 4 catalog. RHEA-2024:0854 The NBDE Tang Server Operator 1.0.1 has been moved from the alpha channel to the stable channel. RHBA-2024:8681 The 1.0.2 update contains fixes that increase the Container Health Index of containers deployed with the NBDE Tang Server Operator to the highest grade. RHEA-2024:10970 The 1.0.3 update contains changes that re-increase the Container Health Index to the highest grade. RHBA-2025:0663 With the NBDE Tang Server Operator 1.1, the golang package is provided in version 1.23.2 and the golang.org/x/net/html package has been updated to version 0.33.0. The updates increase the Container Health Index. 8.3. Understanding the NBDE Tang Server Operator You can use the NBDE Tang Server Operator to automate the deployment of a Tang server in an OpenShift Container Platform cluster that requires Network Bound Disk Encryption (NBDE) internally, leveraging the tools that OpenShift Container Platform provides to achieve this automation. The NBDE Tang Server Operator simplifies the installation process and uses native features provided by the OpenShift Container Platform environment, such as multi-replica deployment, scaling, traffic load balancing, and so on. The Operator also provides automation of certain operations that are error-prone when you perform them manually, for example: server deployment and configuration key rotation hidden keys deletion The NBDE Tang Server Operator is implemented using the Operator SDK and allows the deployment of one or more Tang servers in OpenShift through custom resource definitions (CRDs). 8.3.1. Additional resources Tang-Operator: Providing NBDE in OpenShift Red Hat Hybrid Cloud blog article tang-operator Github project Configuring automated unlocking of encrypted volumes using policy-based decryption chapter in the RHEL 9 Security hardening document 8.4. Installing the NBDE Tang Server Operator You can install the NBDE Tang Operator either by using the web console or through the oc command from CLI. 8.4.1. Installing the NBDE Tang Server Operator using the web console You can install the NBDE Tang Server Operator from the OperatorHub using the web console. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. Procedure In the OpenShift Container Platform web console, navigate to Operators OperatorHub . Search for the NBDE Tang Server Operator: Click Install . On the Operator Installation screen, keep the Update channel , Version , Installation mode , Installed Namespace , and Update approval fields on the default values. After you confirm the installation options by clicking Install , the console displays the installation confirmation. Verification Navigate to the Operators Installed Operators page. Check that the NBDE Tang Server Operator is installed and its status is Succeeded . 8.4.2. Installing the NBDE Tang Server Operator using CLI You can install the NBDE Tang Server Operator from the OperatorHub using the CLI. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You have installed the OpenShift CLI ( oc ). Procedure Use the following command to list available Operators on OperatorHub, and limit the output to Tang-related results: USD oc get packagemanifests -n openshift-marketplace | grep tang Example output tang-operator Red Hat In this case, the corresponding packagemanifest name is tang-operator . Create a Subscription object YAML file to subscribe a namespace to the NBDE Tang Server Operator, for example, tang-operator.yaml : Example subscription YAML for tang-operator apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: tang-operator namespace: openshift-operators spec: channel: stable 1 installPlanApproval: Automatic name: tang-operator 2 source: redhat-operators 3 sourceNamespace: openshift-marketplace 4 1 Specify the channel name from where you want to subscribe the Operator. 2 Specify the name of the Operator to subscribe to. 3 Specify the name of the CatalogSource that provides the Operator. 4 The namespace of the CatalogSource. Use openshift-marketplace for the default OperatorHub CatalogSources. Apply the Subscription to the cluster: USD oc apply -f tang-operator.yaml Verification Check that the NBDE Tang Server Operator controller runs in the openshift-operators namespace: USD oc -n openshift-operators get pods Example output NAME READY STATUS RESTARTS AGE tang-operator-controller-manager-694b754bd6-4zk7x 2/2 Running 0 12s 8.5. Configuring and managing Tang servers using the NBDE Tang Server Operator With the NBDE Tang Server Operator, you can deploy and quickly configure Tang servers. On the deployed Tang servers, you can list existing keys and rotate them. 8.5.1. Deploying a Tang server using the NBDE Tang Server Operator You can deploy and quickly configure one or more Tang servers using the NBDE Tang Server Operator in the web console. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You have installed the NBDE Tang Server Operator on your OCP cluster. Procedure In the OpenShift Container Platform web console, navigate to Operators OperatorHub . Select Project , and click Create Project : On the Create Project page, fill in the required information, for example: Click Create . NBDE Tang Server replicas require a Persistent Volume Claim (PVC) for storing encryption keys. In the web console, navigate to Storage PersistentVolumeClaims : On the following PersistentVolumeClaims screen, click Create PersistentVolumeClaim . On the Create PersistentVolumeClaim page, select a storage that fits your deployment scenario. Consider how often you want to rotate the encryption keys. Name your PVC and choose the claimed storage capacity, for example: Navigate to Operators Installed Operators , and click NBDE Tang Server . Click Create instance . On the Create TangServer page, choose the name of the Tang Server instance, amount of replicas, and specify the name of the previously created Persistent Volume Claim, for example: After you enter the required values a change settings that differ from the default values in your scenario, click Create . 8.5.2. Rotating keys using the NBDE Tang Server Operator With the NBDE Tang Server Operator, you also can rotate your Tang server keys. The precise interval at which you should rotate them depends on your application, key sizes, and institutional policy. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You deployed a Tang server using the NBDE Tang Server Operator on your OpenShift cluster. You have installed the OpenShift CLI ( oc ). Procedure List the existing keys on your Tang server, for example: USD oc -n nbde describe tangserver Example output ... Status: Active Keys: File Name: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg Generated: 2022-02-08 15:44:17.030090484 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ... Create a YAML file for moving your active keys to hidden keys, for example, minimal-keyretrieve-rotate-tangserver.yaml : Example key-rotation YAML for tang-operator apiVersion: daemons.redhat.com/v1alpha1 kind: TangServer metadata: name: tangserver namespace: nbde finalizers: - finalizer.daemons.tangserver.redhat.com spec: replicas: 1 hiddenKeys: - sha1: "PvYQKtrTuYsMV2AomUeHrUWkCGg" 1 1 Specify the SHA-1 thumbprint of your active key to rotate it. Apply the YAML file: USD oc apply -f minimal-keyretrieve-rotate-tangserver.yaml Verification After a certain amount of time depending on your configuration, check that the activeKey value is the new hiddenKey value and the activeKey key file is newly generated, for example: USD oc -n nbde describe tangserver Example output ... Spec: Hidden Keys: sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg Replicas: 1 Status: Active Keys: File Name: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY.jwk Generated: 2023-10-25 15:38:18.134939752 +0000 sha1: vVxkNCNq7gygeeA9zrHrbc3_NZ4 sha256: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY Hidden Keys: File Name: .QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg.jwk Generated: 2023-10-25 15:37:29.126928965 +0000 Hidden: 2023-10-25 15:38:13.515467436 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ... 8.5.3. Deleting hidden keys with the NBDE Tang Server Operator After you rotate your Tang server keys, the previously active keys become hidden and are no longer advertised by the Tang instance. You can use the NBDE Tang Server Operator to remove encryption keys no longer used. WARNING Do not remove any hidden keys unless you are sure that all bound Clevis clients already use new keys. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You deployed a Tang server using the NBDE Tang Server Operator on your OpenShift cluster. You have installed the OpenShift CLI ( oc ). Procedure List the existing keys on your Tang server, for example: USD oc -n nbde describe tangserver Example output ... Status: Active Keys: File Name: PvYQKtrTuYsMV2AomUeHrUWkCGg.jwk Generated: 2022-02-08 15:44:17.030090484 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ... Create a YAML file for removing all hidden keys, for example, hidden-keys-deletion-tangserver.yaml : Example hidden-keys-deletion YAML for tang-operator apiVersion: daemons.redhat.com/v1alpha1 kind: TangServer metadata: name: tangserver namespace: nbde finalizers: - finalizer.daemons.tangserver.redhat.com spec: replicas: 1 hiddenKeys: [] 1 1 The empty array as the value of the hiddenKeys entry indicates you want to preserve no hidden keys on your Tang server. Apply the YAML file: USD oc apply -f hidden-keys-deletion-tangserver.yaml Verification After a certain amount of time depending on your configuration, check that the active key still exists, but no hidden key is available, for example: USD oc -n nbde describe tangserver Example output ... Spec: Hidden Keys: sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg Replicas: 1 Status: Active Keys: File Name: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY.jwk Generated: 2023-10-25 15:38:18.134939752 +0000 sha1: vVxkNCNq7gygeeA9zrHrbc3_NZ4 sha256: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY Status: Ready: 1 Running: 1 Service External URL: http://35.222.247.84:7500/adv Tang Server Error: No Events: ... 8.6. Identifying URL of a Tang server deployed with the NBDE Tang Server Operator Before you can configure your Clevis clients to use encryption keys advertised by your Tang servers, you must identify the URLs of the servers. 8.6.1. Identifying URL of the NBDE Tang Server Operator using the web console You can identify the URLs of Tang servers deployed with the NBDE Tang Server Operator from the OperatorHub by using the OpenShift Container Platform web console. After you identify the URLs, you use the clevis luks bind command on your clients containing LUKS-encrypted volumes that you want to unlock automatically by using keys advertised by the Tang servers. See the Configuring manual enrollment of LUKS-encrypted volumes section in the RHEL 9 Security hardening document for detailed steps describing the configuration of clients with Clevis. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You deployed a Tang server by using the NBDE Tang Server Operator on your OpenShift cluster. Procedure In the OpenShift Container Platform web console, navigate to Operators Installed Operators Tang Server . On the NBDE Tang Server Operator details page, select Tang Server . The list of Tang servers deployed and available for your cluster appears. Click the name of the Tang server you want to bind with a Clevis client. The web console displays an overview of the selected Tang server. You can find the URL of your Tang server in the Tang Server External Url section of the screen: In this example, the URL of the Tang server is http://34.28.173.205:7500 . Verification You can check that the Tang server is advertising by using curl , wget , or similar tools, for example: USD curl 2> /dev/null http://34.28.173.205:7500/adv | jq Example output { "payload": "eyJrZXlzIj...eSJdfV19", "protected": "eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9", "signature": "AUB0qSFx0FJLeTU...aV_GYWlDx50vCXKNyMMCRx" } 8.6.2. Identifying URL of the NBDE Tang Server Operator using CLI You can identify the URLs of Tang servers deployed with the NBDE Tang Server Operator from the OperatorHub by using the CLI. After you identify the URLs, you use the clevis luks bind command on your clients containing LUKS-encrypted volumes that you want to unlock automatically by using keys advertised by the Tang servers. See the Configuring manual enrollment of LUKS-encrypted volumes section in the RHEL 9 Security hardening document for detailed steps describing the configuration of clients with Clevis. Prerequisites You must have cluster-admin privileges on an OpenShift Container Platform cluster. You have installed the OpenShift CLI ( oc ). You deployed a Tang server by using the NBDE Tang Server Operator on your OpenShift cluster. Procedure List details about your Tang server, for example: USD oc -n nbde describe tangserver Example output ... Spec: ... Status: Ready: 1 Running: 1 Service External URL: http://34.28.173.205:7500/adv Tang Server Error: No Events: ... Use the value of the Service External URL: item without the /adv part. In this example, the URL of the Tang server is http://34.28.173.205:7500 . Verification You can check that the Tang server is advertising by using curl , wget , or similar tools, for example: USD curl 2> /dev/null http://34.28.173.205:7500/adv | jq Example output { "payload": "eyJrZXlzIj...eSJdfV19", "protected": "eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9", "signature": "AUB0qSFx0FJLeTU...aV_GYWlDx50vCXKNyMMCRx" } 8.6.3. Additional resources Configuring manual enrollment of LUKS-encrypted volumes section in the RHEL 9 Security hardening document.

Commands:
oc get packagemanifests -n openshift-marketplace | grep tang
tang-operator Red Hat
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: tang-operator namespace: openshift-operators spec: channel: stable 1 installPlanApproval: Automatic name: tang-operator 2 source: redhat-operators 3 sourceNamespace: openshift-marketplace 4
oc apply -f tang-operator.yaml
oc -n openshift-operators get pods
NAME READY STATUS RESTARTS AGE tang-operator-controller-manager-694b754bd6-4zk7x 2/2 Running 0 12s
oc -n nbde describe tangserver
... Status: Active Keys: File Name: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg Generated: 2022-02-08 15:44:17.030090484 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ...
apiVersion: daemons.redhat.com/v1alpha1 kind: TangServer metadata: name: tangserver namespace: nbde finalizers: - finalizer.daemons.tangserver.redhat.com spec: replicas: 1 hiddenKeys: - sha1: "PvYQKtrTuYsMV2AomUeHrUWkCGg" 1
oc apply -f minimal-keyretrieve-rotate-tangserver.yaml
oc -n nbde describe tangserver
... Spec: Hidden Keys: sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg Replicas: 1 Status: Active Keys: File Name: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY.jwk Generated: 2023-10-25 15:38:18.134939752 +0000 sha1: vVxkNCNq7gygeeA9zrHrbc3_NZ4 sha256: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY Hidden Keys: File Name: .QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg.jwk Generated: 2023-10-25 15:37:29.126928965 +0000 Hidden: 2023-10-25 15:38:13.515467436 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ...
oc -n nbde describe tangserver
... Status: Active Keys: File Name: PvYQKtrTuYsMV2AomUeHrUWkCGg.jwk Generated: 2022-02-08 15:44:17.030090484 +0000 sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg sha256: QS82aXnPKA4XpfHr3umbA0r2iTbRcpWQ0VI2Qdhi6xg ...
apiVersion: daemons.redhat.com/v1alpha1 kind: TangServer metadata: name: tangserver namespace: nbde finalizers: - finalizer.daemons.tangserver.redhat.com spec: replicas: 1 hiddenKeys: [] 1
oc apply -f hidden-keys-deletion-tangserver.yaml
oc -n nbde describe tangserver
... Spec: Hidden Keys: sha1: PvYQKtrTuYsMV2AomUeHrUWkCGg Replicas: 1 Status: Active Keys: File Name: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY.jwk Generated: 2023-10-25 15:38:18.134939752 +0000 sha1: vVxkNCNq7gygeeA9zrHrbc3_NZ4 sha256: T-0wx1HusMeWx4WMOk4eK97Q5u4dY5tamdDs7_ughnY Status: Ready: 1 Running: 1 Service External URL: http://35.222.247.84:7500/adv Tang Server Error: No Events: ...
curl 2> /dev/null http://34.28.173.205:7500/adv | jq
payload: eyJrZXlzIj...eSJdfV19
protected: eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9
signature: AUB0qSFx0FJLeTU...aV_GYWlDx50vCXKNyMMCRx
oc -n nbde describe tangserver
... Spec: ... Status: Ready: 1 Running: 1 Service External URL: http://34.28.173.205:7500/adv Tang Server Error: No Events: ...
curl 2> /dev/null http://34.28.173.205:7500/adv | jq
payload: eyJrZXlzIj...eSJdfV19
protected: eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9
signature: AUB0qSFx0FJLeTU...aV_GYWlDx50vCXKNyMMCRx