Chapter 11. Managing secrets and keys with OpenStack Key Manager (barbican) The OpenStack Key Manager (barbican) service for Red Hat OpenStack Services on OpenShift (RHOSO) securely stores, provisions and manages your secret data. You use OpenStack Key Manager to create, update, and delete secrets and encryption keys. 11.1. About the Red Hat OpenStack Services on OpenShift Key Manager Operator (barbican) The OpenStack Key Manager (barbican) service is the secrets manager for Red Hat OpenStack Services on OpenShift (RHOSO). This Operator is installed and active by default. An OpenStack administrator can use the the Key Manager API and command line to centrally manage the certificates, keys, and passwords used by OpenStack. The RHOSO Key Manager service integrates with Block Storage (cinder), Networking (neutron), and Compute (nova) services. Secrets such as certificates, API keys, and passwords, are stored in the barbican database or directly in a secure storage system. In RHOSO 18.0. the simple crypto plug-in is available to encrypt secrets. OpenStack Key Manager supports the following use cases: Symmetric encryption keys Block Storage (cinder) volume encryption Object Storage (swift) encryption Ephemeral disk encryption Asymmetric keys and certificates Image service (glance) image signing Image service (glance) image verification 11.2. Viewing secrets To view the list of secrets, run the openstack secret list command. The list includes the URI, name, type, and other information about the secrets. Procedure View the list of secrets: Example command Example output 11.3. Creating a secret To create a secret, run the openstack secret store command and specify the name of the secret and optionally the payload for the secret. Procedure Create a secret: Example command Example Output 11.4. Adding a payload to a secret You cannot change the payload of a secret, you can only delete it. If you create a secret without specifying a payload, then you can later add a payload to it by using the openstack secret update command. Procedure Add a payload to a secret: Example command 11.5. Deleting a secret To delete a secret, run the openstack secret delete command and specify the secret URI. Procedure Delete a secret with the specified URI: Example command 11.6. Generating a symmetric key You can generate a symmetric key for specific tasks such as Compute service disk encryption and Object Storage service encryption. Procedure Generate a new 256-bit key using order create and store it in barbican. For example: Example command Example output You can also use the --mode option to configure generated keys to use a particular mode, such as ctr or cbc . For more information, see NIST SP 800-38A . View the details of the order to identify the location of the generated key, shown here as the Secret href value: Example command Example output Retrieve the details of the secret: Example command Example output 11.7. Encrypting Block Storage (cinder) volumes You can use barbican to manage your Block Storage service (cinder) encryption keys. This configuration uses LUKS to encrypt the disks attached to your instances, including boot disks. Key management is transparent to the user; when you create a new volume using luks as the encryption type, cinder generates a symmetric key secret for the volume and stores it in barbican. When booting the instance or attaching an encrypted volume, the Compute service retrieves the key from the Key Manager service and stores the secret locally as a libvirt secret on the Compute node. Procedure Create a volume template that uses encryption. When you create new volumes, they can be modeled off the settings you define here: Example command Example output Create a new volume and specify that it uses the LuksEncryptor-Template-256 settings: Example command Example output The resulting secret is automatically uploaded to the barbican back end. Ensure that you can see the barbican secret UUID. This value is displayed in the encryption_key_id field. Example command Example output Note You must use the --os-volume-api-version 3.64 parameter with the Cinder CLI to display the encryption_key_id value. There is no equivalent OpenStack CLI command. Use barbican to confirm that the disk encryption key is present. In this example, the timestamp matches the LUKS volume creation time: Example command Example output Attach the new volume to an existing instance: The volume is now available for you to mount to the guest file system. 11.8. Signing Image Service (glance) images When you configure the Image Service (glance) to validate images, you must sign those images before uploading them. Use the openssl command to sign an image with a key that is stored in the Key Manager service (barbican), and then upload the image to the Image service with the accompanying image properties. The signature of the image is verified before each use, and the instance build process fails if the signature does not match. Limitations Signature verification is based on openSSL _rsa_sig_verify() , which currently raises an InvalidSignature exception in the Image service if the key length is not 1024. This issue occurs because of a regression in OpenSSL 3.0.7. Signature verification works correctly with key length 2048 and OpenSSL 3.0.2. OpenSSL does not support the SHA256 algorithm. You must use SHA512 or greater to create a signed image. Procedure Use the openssl command to create the keys and the certificate: Create the private key: Create the public key: Create the certificate: When prompted, enter details to include in your certificate request: Upload the certificate to the Key Manager service, where it is stored in the secret store : Example output: Note Record the certificate's UUID for use in a later step. In this example, the UUID is cd7cc675-e573-419c-8fff-33a72734a243 . Retrieve an image and create a signature: Retrieve an image: Use private_key.pem to sign the image and generate the .signature file. For example: Convert the .signature file to Base64 format: Load the Base64 value into a variable to use when you upload the image: Upload an image with valid signature properties to the Image service: 11.9. Validating snapshots Snapshots are saved as Image service (glance) images. If you configure the Compute service (nova) to check for signed images, then snapshots must by signed, even if they were created from an instance with a signed image. Procedure Download the snapshot from the Image service: Replace <local_file_name> with the file name of the downloaded image. Replace <snapshot_image_name> with the name of your snapshot. Generate a signature to validate the snapshot. Use the same process that you use to generate a signature to validate an image. For more information, see Signing Image service (glance) images . Update the snapshot image properties: Replace <cd7cc675-e573-419c-8fff-33a72734a243> with the UUID of the certificate for the image. Replace <snapshot_image_id> with the ID of your snapshot. Optional: Remove the downloaded Image service image from the filesystem: 11.10. Backing up simple crypto encryption keys To backup simple crypto encryption keys, you must back up the Key Manager (barbican) database and the BarbicanSimpleCryptoKek key that encrypts and decrypts values from the database. Store the key and Key Manager database in a security-hardened location. Prerequisites You have a security-hardened location for the BarbicanSimpleCryptoKEK key and Key Manager database. Procedure Retrieve the base64 decoded BarbicanSimpleCryptoKEK value from osp-secret and save it to a file called kek.txt: Retrieve the Key Manager database root password: Use the Key Manager database root password to back up the barbican database: Check that the database backup is stored in your current working directory: Back up the barbican_db_backup.sql and kek.txt files to a security hardened location. 11.11. Restoring simple crypto encryption keys Restore the barbican database and the BarbicanSimpleCryptoKek password from a backup in order to restore simple crypto encryption keys. Procedure Encode your BarbicanSimpleCryptoKek key using base64 and write the resulting value into osp-secret: Replace <barbican_key> with the BarbicanSimpleCryptoKek key that you want to restore Open the osp-secret secret object for editing: Add the value from step 1. Write and save your changes: Retrieve the barbican database root password: Restore the backup file to the barbican database: Replace <sql_backup> with the name of your mysql database backup file. Example output Verification You can use the openstackclient pod to verify that the test secrets were restored successfully:

Commands:
openstack secret list
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://192.168.123.169:9311/v1/secrets/24845e6d-64a5-4071-ba99-0fdd1046172e | None | 2018-01-22T02:23:15+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | +------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
openstack secret store --name testSecret --payload 'TestPayload'
+---------------+------------------------------------------------------------------------------------+ | Field | Value | +---------------+------------------------------------------------------------------------------------+ | Secret href | https://192.168.123.163:9311/v1/secrets/ecc7b2a4-f0b0-47ba-b451-0f7d42bc1746 | | Name | testSecret | | Created | None | | Status | None | | Content types | None | | Algorithm | aes | | Bit length | 256 | | Secret type | opaque | | Mode | cbc | | Expiration | None | +---------------+------------------------------------------------------------------------------------+
openstack secret update https://192.168.123.163:9311/v1/secrets/ca34a264-fd09-44a1-8856-c6e7116c3b16 'TestPayload-updated'
openstack secret delete https://192.168.123.163:9311/v1/secrets/ecc7b2a4-f0b0-47ba-b451-0f7d42bc1746
openstack secret order create --name swift_key --algorithm aes --mode ctr --bit-length 256 --payload-content-type=application/octet-stream key
+----------------+-----------------------------------------------------------------------------------+ | Field | Value | +----------------+-----------------------------------------------------------------------------------+ | Order href | https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832 | | Type | Key | | Container href | N/A | | Secret href | None | | Created | None | | Status | None | | Error code | None | | Error message | None | +----------------+-----------------------------------------------------------------------------------+
openstack secret order get https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832
+----------------+------------------------------------------------------------------------------------+ | Field | Value | +----------------+------------------------------------------------------------------------------------+ | Order href | https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832 | | Type | Key | | Container href | N/A | | Secret href | https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719 | | Created | 2018-01-24T04:24:33+00:00 | | Status | ACTIVE | | Error code | None | | Error message | None | +----------------+------------------------------------------------------------------------------------+
openstack secret get https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719
+---------------+------------------------------------------------------------------------------------+ | Field | Value | +---------------+------------------------------------------------------------------------------------+ | Secret href | https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719 | | Name | swift_key | | Created | 2018-01-24T04:24:33+00:00 | | Status | ACTIVE | | Content types | {u'default': u'application/octet-stream'} | | Algorithm | aes | | Bit length | 256 | | Secret type | symmetric | | Mode | ctr | | Expiration | None | +---------------+------------------------------------------------------------------------------------+
openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | description | None | | encryption | cipher='aes-xts-plain64', control_location='front-end', encryption_id='9df604d0-8584-4ce8-b450-e13e6316c4d3', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor' | | id | 78898a82-8f4c-44b2-a460-40a5da9e4d59 | | is_public | True | | name | LuksEncryptor-Template-256 | +-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
openstack volume create --size 1 --type LuksEncryptor-Template-256 'Encrypted-Test-Volume'
+---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2018-01-22T00:19:06.000000 | | description | None | | encrypted | True | | id | a361fd0b-882a-46cc-a669-c633630b5c93 | | migration_status | None | | multiattach | False | | name | Encrypted-Test-Volume | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | LuksEncryptor-Template-256 | | updated_at | None | | user_id | 0e73cb3111614365a144e7f8f1a972af | +---------------------+--------------------------------------+
cinder --os-volume-api-version 3.64 volume show Encrypted-Test-Volume
+------------------------------+-------------------------------------+ |Property |Value | +------------------------------+-------------------------------------+ |attached_servers |[] | |attachment_ids |[] | |availability_zone |nova | |bootable |false | |cluster_name |None | |consistencygroup_id |None | |created_at |2022-07-28T17:35:26.000000 | |description |None | |encrypted |True | |encryption_key_id |0944b8a8-de09-4413-b2ed-38f6c4591dd4 | |group_id |None | |id |a0b51b97-0392-460a-abfa-093022a120f3 | |metadata | | |migration_status |None | |multiattach |False | |name |vol | |os-vol-host-attr:host |hostgroup@tripleo_iscsi#tripleo_iscsi| |os-vol-mig-status-attr:migstat|None | |os-vol-mig-status-attr:name_id|None | |os-vol-tenant-attr:tenant_id |a2071ece39b3440aa82395ff7707996f | |provider_id |None | |replication_status |None | |service_uuid |471f0805-072e-4256-b447-c7dd10ceb807 | |shared_targets |False | |size |1 | |snapshot_id |None | |source_volid |None | |status |available | |updated_at |2022-07-28T17:35:26.000000 | |user_id |ba311b5c2b8e438c951d1137333669d4 | |volume_type |LUKS | |volume_type_id |cc188ace-f73d-4af5-bf5a-d70ccc5a401c | +------------------------------+-------------------------------------+
openstack secret list
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://192.168.123.169:9311/v1/secrets/0944b8a8-de09-4413-b2ed-38f6c4591dd4 | None | 2018-01-22T02:23:15+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | +------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
openstack server add volume testInstance Encrypted-Test-Volume
openssl genrsa -out private_key.pem 1024
openssl rsa -pubout -in private_key.pem -out public_key.pem
openssl req -new -key private_key.pem -out cert_request.csr
openssl x509 -req -days 14 -in cert_request.csr -signkey private_key.pem -out new_cert.crt
openstack secret store --name test --algorithm RSA --secret-type certificate --payload-content-type "application/octet-stream" --payload-content-encoding base64 --payload "USD(base64 new_cert.crt)"
+---------------+-----------------------------------------------------------------------+ | Field | Value | +---------------+-----------------------------------------------------------------------+ | Secret href | http://127.0.0.1:9311/v1/secrets/cd7cc675-e573-419c-8fff-33a72734a243 | +---------------+-----------------------------------------------------------------------+
echo <This is my image> > <myimage>
openssl dgst -sha512 -sign private_key.pem -sigopt rsa_padding_mode:pss -out myimage.signature myimage
base64 -w 0 myimage.signature > myimage.signature.b64
image_signature=USD(cat myimage.signature.b64)
openstack image-create --name <my_signed_image> --container-format bare --disk-format qcow2 --property img_signature="USDimage_signature" --property img_signature_certificate_uuid="USDcert_uuid" --property img_signature_hash_method='SHA-512' --property img_signature_key_type='RSA-PSS' < myimage
openstack image save --file <local_file_name> <snapshot_image_name>
openstack image set --property img_signature="USDimage_signature" --property img_signature_certificate_uuid="<cd7cc675-e573-419c-8fff-33a72734a243>" --property img_signature_hash_method="SHA-512" --property img_signature_key_type="RSA-PSS" <snapshot_image_id>
rm <local_file_name>
get secret osp-secret -o yaml | grep BarbicanSimpleCryptoKEK | awk '{print USD2}' | base64 -d > kek.txt
DB_PASS=USD(oc get secret osp-secret -o yaml | grep DbRootPassword | awk '{print USD2}' | base64 -d)
oc exec -it openstack-galera-0 -- mysqldump -u root -p"USD{DB_PASS}" barbican > barbican_db_backup.sql
ll total 36 -rw-rw-r--. 1 tripleo-admin tripleo-admin 36715 Jun 19 18:31 barbican_db_backup.sql
echo <barbican_key> | base64 YmFyYmljYW5fc2ltcGxlX2N5cHRvX2tlawo=
oc edit secret osp-secret
. BarbicanDatabasePassword: cGFzc3dvcmQK BarbicanPassword: cGFzc3dvcmQK BarbicanSimpleCryptoKEK: YmFyYmljYW5fc2ltcGxlX2N5cHRvX2tlawo= CeilometerPassword: cGFzc3dvcmQK CinderDatabasePassword: cGFzc3dvcmQK CinderPassword: cGFzc3dvcmQK DatabasePassword: cGFzc3dvcmQK .
DB_PASS=USD(oc get secret osp-secret -o yaml | grep DbRootPassword | awk '{print USD2}' | base64 -d)
oc exec openstack-galera-0 -- mysql -u root -p"USD{DB_PASS}" barbican < <sql_backup>
Defaulted container "galera" out of: galera, mysql-bootstrap (init)
oc exec -n openstack -t openstackclient -- openstack secret list +------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.104:9311/v1/secrets/93f62cfd-e008-401f-be74-bf057c88b04a | testSecret | 2018-06-19T18:25:25+00:00 | ACTIVE | {u'default': u'text/plain'} | aes | 256 | opaque | cbc | None | | http://10.0.0.104:9311/v1/secrets/f664b5cf-5221-47e5-9887-608972a5fefb | swift_key | 2018-06-19T18:24:40+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | ctr | None | +------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+