Chapter 8. Security As a storage administrator, securing the storage cluster environment is important. Red Hat Ceph Storage provides encryption and key management to secure the Ceph Object Gateway access point. Prerequisites A healthy running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. 8.1. Server-Side Encryption (SSE) The Ceph Object Gateway supports server-side encryption of uploaded objects for the S3 application programming interface (API). Server-side encryption means that the S3 client sends data over HTTP in its unencrypted form, and the Ceph Object Gateway stores that data in the Red Hat Ceph Storage cluster in encrypted form. Note Red Hat does NOT support S3 object encryption of Static Large Object (SLO) or Dynamic Large Object (DLO). Currently, none of the Server-Side Encryption (SSE) modes have implemented support for CopyObject . It is currently being developed [BZ#2149450 ]. Important Server-side encryption is not compatible with multi-site replication due to a known issue. This issue will be resolved in a future release. See Known issues- Mult-site Object Gateway for more details. Important To use encryption, client requests MUST send requests over an SSL connection. Red Hat does not support S3 encryption from a client unless the Ceph Object Gateway uses SSL. However, for testing purposes, administrators can disable SSL during testing by setting the rgw_crypt_require_ssl configuration setting to false at runtime, using the ceph config set client.rgw command, and then restarting the Ceph Object Gateway instance. In a production environment, it might not be possible to send encrypted requests over SSL. In such a case, send requests using HTTP with server-side encryption. For information about how to configure HTTP with server-side encryption, see the Additional Resources section below. There are three options for the management of encryption keys: Customer-provided Keys When using customer-provided keys, the S3 client passes an encryption key along with each request to read or write encrypted data. It is the customer's responsibility to manage those keys. Customers must remember which key the Ceph Object Gateway used to encrypt each object. Ceph Object Gateway implements the customer-provided key behavior in the S3 API according to the Amazon SSE-C specification. Since the customer handles the key management and the S3 client passes keys to the Ceph Object Gateway, the Ceph Object Gateway requires no special configuration to support this encryption mode. Key Management Service When using a key management service, the secure key management service stores the keys and the Ceph Object Gateway retrieves them on demand to serve requests to encrypt or decrypt data. Ceph Object Gateway implements the key management service behavior in the S3 API according to the Amazon SSE-KMS specification. Important Currently, the only tested key management implementations are HashiCorp Vault, and OpenStack Barbican. However, OpenStack Barbican is a Technology Preview and is not supported for use in production systems. SSE-S3 When using SSE-S3, the keys are stored in vault, but they are automatically created and deleted by Ceph and retrieved as required to serve requests to encrypt or decrypt data. Ceph Object Gateway implements the SSE-S3 behavior in the S3 API according to the Amazon SSE-S3 specification. Additional Resources Amazon SSE-C Amazon SSE-KMS Configuring server-side encryption The HashiCorp Vault 8.1.1. Setting the default encryption for an existing S3 bucket As a storage administrator, you can set the default encryption for an existing Amazon S3 bucket so that all objects are encrypted when they are stored in a bucket. You can use Bucket Encryption APIs to support server-side encryption with Amazon S3-managed keys (SSE-S3) or Amazon KMS customer master keys (SSE-KMS). Note SSE-KMS is supported only from Red Hat Ceph Storage 5.x, not for Red Hat Ceph Storage 4.x. You can manage default encryption for an existing Amazon S3 bucket using the PutBucketEncryption API. All files uploaded to this bucket will have this encryption by defining the default encryption at the bucket level. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway. An S3 bucket created. An S3 user created with user access. Access to a Ceph Object Gateway client with the AWS CLI package installed. Procedure Create a JSON file for the encryption configuration: Example Add the encryption configuration rules to the file: Example Set the default encryption for the bucket: Syntax Example Verification Retrieve the bucket encryption configuration for the bucket: Syntax Example Note If the bucket does not have a default encryption configuration, the get-bucket-encryption command returns ServerSideEncryptionConfigurationNotFoundError . 8.1.2. Deleting the default bucket encryption You can delete the default bucket encryption for a specified bucket using the s3api delete-bucket-encryption command. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway. An S3 bucket created. An S3 user created with user access. Access to a Ceph Object Gateway client with the AWS CLI package installed. Procedure Delete a bucket encryption configuration: Syntax Example Verification Retrieve the bucket encryption configuration for the bucket: Syntax Example 8.2. Server-side encryption requests In a production environment, clients often contact the Ceph Object Gateway through a proxy. This proxy is referred to as a load balancer because it connects to multiple Ceph Object Gateways. When the client sends requests to the Ceph Object Gateway, the load balancer routes those requests to the multiple Ceph Object Gateways, thus distributing the workload. In this type of configuration, it is possible that SSL terminations occur both at a load balancer and between the load balancer and the multiple Ceph Object Gateways. Communication occurs using HTTP only. To set up the Ceph Object Gateways to accept the server-side encryption requests, see Configuring server-side encryption . 8.3. Configuring server-side encryption You can set up server-side encryption to send requests to the Ceph Object Gateway using HTTP, in cases where it might not be possible to send encrypted requests over SSL. This procedure uses HAProxy as proxy and load balancer. Prerequisites Root-level access to all nodes in the storage cluster. A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. Installation of the HAProxy software. Procedure Edit the haproxy.cfg file: Example Comment out the lines that allow access to the http front end and add instructions to direct HAProxy to use the https front end instead: Example Set the rgw_trust_forwarded_https option to true : Example Enable and start HAProxy: Additional Resources See the High availability service section in the Red Hat Ceph Storage Object Gateway Guide for additional details. See the Red Hat Ceph Storage installation chapter in the Red Hat Ceph Storage Installation Guide for additional details. 8.4. The HashiCorp Vault As a storage administrator, you can securely store keys, passwords, and certificates in the HashiCorp Vault for use with the Ceph Object Gateway. The HashiCorp Vault provides a secure key management service for server-side encryption used by the Ceph Object Gateway. The basic workflow: The client requests the creation of a secret key from the Vault based on an object's key ID. The client uploads an object with the object's key ID to the Ceph Object Gateway. The Ceph Object Gateway then requests the newly created secret key from the Vault. The Vault replies to the request by returning the secret key to the Ceph Object Gateway. Now the Ceph Object Gateway can encrypt the object using the new secret key. After encryption is done the object is then stored on the Ceph OSD. Important Red Hat works with our technology partners to provide this documentation as a service to our customers. However, Red Hat does not provide support for this product. If you need technical assistance for this product, then contact Hashicorp for support. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. Installation of the HashiCorp Vault software. 8.4.1. Secret engines for Vault The HashiCorp Vault provides several secret engines to generate, store, or encrypt data. The application programming interface (API) sends data calls to the secret engine asking for action on that data, and the secret engine returns a result of that action request. The Ceph Object Gateway supports two of the HashiCorp Vault secret engines: Key/Value version 2 Transit Important The secret engines can be configured at any time but the engines are not supported simultaneously. Key/Value version 2 The Key/Value secret engine stores random secrets within the Vault, on disk. With version 2 of the kv engine, a key can have a configurable number of versions. The default number of versions is 10. Deleting a version does not delete the underlying data, but marks the data as deleted, allowing deleted versions to be undeleted. You can use the API endpoint or the destroy command to permanently remove a version's data. To delete all versions and metadata for a key, you can use the metadata command or the API endpoint. The key names must be strings, and the engine will convert non-string values into strings when using the command line interface. To preserve non-string values, provide a JSON file or use the HTTP application programming interface (API). Note For access control list (ACL) policies, the Key/Value secret engine recognizes the distinctions between the create and update capabilities. Transit The Transit secret engine performs cryptographic functions on in-transit data. The Transit secret engine can generate hashes, can be a source of random bytes, and can also sign and verify data. The Vault does not store data when using the Transit secret engine. The Transit secret engine supports key derivation, by allowing the same key to be used for multiple purposes. Also, the transit secret engine supports key versioning. The Transit secret engine supports these key types: aes128-gcm96 AES-GCM with a 128-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption aes256-gcm96 AES-GCM with a 256-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption (default) chacha20-poly1305 ChaCha20-Poly1305 with a 256-bit key; supports encryption, decryption, key derivation, and convergent encryption ed25519 Ed25519; supports signing, signature verification, and key derivation ecdsa-p256 ECDSA using curve P-256; supports signing and signature verification ecdsa-p384 ECDSA using curve P-384; supports signing and signature verification ecdsa-p521 ECDSA using curve P-521; supports signing and signature verification rsa-2048 2048-bit RSA key; supports encryption, decryption, signing, and signature verification rsa-3072 3072-bit RSA key; supports encryption, decryption, signing, and signature verification rsa-4096 4096-bit RSA key; supports encryption, decryption, signing, and signature verification Additional Resources See the KV Secrets Engine documentation on Vault's project site for more information. See the Transit Secrets Engine documentation on Vault's project site for more information. 8.4.2. Authentication for Vault The HashiCorp Vault supports several types of authentication mechanisms. The Ceph Object Gateway currently supports the Vault agent method. The Ceph Object Gateway uses the rgw_crypt_vault_auth , and rgw_crypt_vault_addr options to configure the use of the HashiCorp Vault. Important Red Hat supports the usage of Vault agent as the authentication method for containers and the usage of token authentication is not supported on containers. Vault Agent The Vault agent is a daemon that runs on a client node and provides client-side caching, along with token renewal. The Vault agent typically runs on the Ceph Object Gateway node. Run the Vault agent and refresh the token file. When the Vault agent is used in this mode, you can use file system permissions to restrict who has access to the usage of tokens. Also, the Vault agent can act as a proxy server, that is, Vault will add a token when required and add it to the requests passed to it before forwarding them to the actual server. The Vault agent can still handle token renewal just as it would when storing a token in the Filesystem. It is required to secure the network that Ceph Object Gateways uses to connect with the Vault agent, for example, the Vault agent listens to only the localhost. Additional Resources See the Vault Agent documentation on Vault's project site for more information. 8.4.3. Namespaces for Vault Using HashiCorp Vault as an enterprise service provides centralized management for isolated namespaces that teams within an organization can use. These isolated namespace environments are known as tenants , and teams within an organization can utilize these tenants to isolate their policies, secrets, and identities from other teams. The namespace features of Vault help support secure multi-tenancy from within a single infrastructure. Additional Resources See the Vault Enterprise Namespaces documentation on Vault's project site for more information. 8.4.4. Transit engine compatibility support There is compatibility support for the versions of Ceph which used the Transit engine as a simple key store. You can use the compat option in the Transit engine to configure the compatibility support. You can disable support with the following command: Example Note This is the default for future versions and you can use the current version for new installations. The normal default with the current version is: Example This enables the new engine for newly created objects and still allows the old engine to be used for the old objects. To access old and new objects, the Vault token must have both the old and new transit policies. You can force use only the old engine with the following command: Example This mode is selected by default if the Vault ends in export/encryption-key . Important After configuring the client.rgw options, you need to restart the Ceph Object Gateway daemons for the new values to take effect. Additional Resources See the Vault Agent documentation on Vault's project site for more information. 8.4.5. Creating token policies for Vault A token policy specifies the powers that all Vault tokens have. One token can have multiple policies. You should use the required policy for the configuration. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the HashiCorp Vault software. Root-level access to the HashiCorp Vault node. Procedure Create a token policy: For the Key/Value secret engine: Example For the Transit engine: Example Note If you have used the Transit secret engine on an older version of Ceph, the token policy is: Example If you are using both SSE-KMS and SSE-S3, you should point each to separate containers. You could either use separate Vault instances or separately mount transit instances or different branches under a common transit point. If you are not using separate Vault instances, you can point SSE-KMS or SSE-S3 to serparate containers using rgw_crypt_vault_prefix and rgw_crypt_sse_s3_vault_prefix . When granting Vault permissions to SSE-KMS bucket owners, you should not give them permission to SSE-S3 keys; only Ceph should have access to the SSE-S3 keys. 8.4.6. Configuring the Ceph Object Gateway to use SSE-KMS with Vault To configure the Ceph Object Gateway to use the HashiCorp Vault with SSE-KMS for key management, it must be set as the encryption key store. Currently, the Ceph Object Gateway supports two different secret engines, and two different authentication methods. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. Root-level access to a Ceph Object Gateway node. Procedure Use the ceph config set client.rgw OPTION VALUE command to enable Vault as the encryption key store: Syntax Add the following options and values: Syntax Customize the policy as per the use case. Get the role-id: Syntax Get the secret-id: Syntax Create the configuration for the Vault agent: Example Use systemctl to run the persistent daemon: Example A token file is populated with a valid token when the Vault agent runs. Select a Vault secret engine, either Key/Value or Transit. If using Key/Value , then add the following line: Example If using Transit , then add the following line: Example Use the ceph config set client.rgw OPTION VALUE command to set the Vault namespace to retrieve the encryption keys: Example Restrict where the Ceph Object Gateway retrieves the encryption keys from the Vault by setting a path prefix: Example For exportable Transit keys, set the prefix path as follows: Example Assuming the domain name of the Vault server is vault-server , the Ceph Object Gateway will fetch encrypted transit keys from the following URL: Example Restart the Ceph Object Gateway daemons. To restart the Ceph Object Gateway on an individual node in the storage cluster: Syntax Example To restart the Ceph Object Gateways on all nodes in the storage cluster: Syntax Example Additional Resources See the Secret engines for Vault section of the Red Hat Ceph Storage Object Gateway Guide for more details. See the Authentication for Vault section of the Red Hat Ceph Storage Object Gateway Guide for more details. 8.4.7. Configuring the Ceph Object Gateway to use SSE-S3 with Vault To configure the Ceph Object Gateway to use the HashiCorp Vault with SSE-S3 for key management, it must be set as the encryption key store. Currently, the Ceph Object Gateway only uses agent authentication method. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. Root-level access to a Ceph Object Gateway node. Procedure Log into the Cephadm shell Example Enable Vault as the secrets engine to retrieve SSE-S3 encryption keys: Syntax To set the authentication method to use with SSE-S3 and Vault, configure the following settings: Syntax Example Customize the policy as per your use case to set up a Vault agent. Get the role-id: Syntax Get the secret-id: Syntax Create the configuration for the Vault agent: Example Use systemctl to run the persistent daemon: Example A token file is populated with a valid token when the Vault agent runs. Set the Vault secret engine to use to retrieve encryption keys, either Key/Value or Transit. If using Key/Value , set the following: Example If using Transit , set the following: Example Optional: Configure the Ceph Object Gateway to access Vault within a particular namespace to retrieve the encryption keys: Example Note Vault namespaces allow teams to operate within isolated environments known as tenants. The Vault namespaces feature is only available in the Vault Enterprise version. Optional: Restrict access to a particular subset of the Vault secret space by setting a URL path prefix, where the Ceph Object Gateway retrieves the encryption keys from: If using Key/Value , set the following: Example If using Transit , set the following: Example Assuming the domain name of the Vault server is vaultserver , the Ceph Object Gateway will fetch encrypted transit keys from the following URL: Example Optional: To use custom SSL certification to authenticate with Vault, configure the following settings: Syntax Example Restart the Ceph Object Gateway daemons. To restart the Ceph Object Gateway on an individual node in the storage cluster: Syntax Example To restart the Ceph Object Gateways on all nodes in the storage cluster: Syntax Example Additional Resources See the Secret engines for Vault section of the Red Hat Ceph Storage Object Gateway Guide for more details. See the Authentication for Vault section of the Red Hat Ceph Storage Object Gateway Guide for more details. 8.4.8. Creating a key using the kv engine Configure the HashiCorp Vault Key/Value secret engine ( kv ) so you can create a key for use with the Ceph Object Gateway. Secrets are stored as key-value pairs in the kv secret engine. Important Keys for server-side encryption must be 256-bits long and encoded using base64 . Prerequisites A running Red Hat Ceph Storage cluster. Installation of the HashiCorp Vault software. Root-level access to the HashiCorp Vault node. Procedure Enable the Key/Value version 2 secret engine: Example Create a new key: Syntax Example 8.4.9. Creating a key using the transit engine Configure the HashiCorp Vault Transit secret engine ( transit ) so you can create a key for use with the Ceph Object Gateway. Creating keys with the Transit secret engine must be exportable in order to be used for server-side encryption with the Ceph Object Gateway. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the HashiCorp Vault software. Root-level access to the HashiCorp Vault node. Procedure Enable the Transit secret engine: Create a new exportable key: Syntax Example Note By default the above command creates a aes256-gcm96 type key. Enable key rotation: Syntax Example Specify the duration for key rotation: Syntax Example In this example, 30d specifies that the key is rotated after 30 days. To specify the key rotation duration in hours, use auto_rotate_period=1h . 1h specifies that the key rotates every 1 hour. Verify key rotation is successful by ensuring that the latest_version value has incremented: Syntax Example Verify the creation of the key: Syntax Example Note Providing the full key path, including the key version, is required. 8.4.10. Uploading an object using AWS and the Vault When uploading an object to the Ceph Object Gateway, the Ceph Object Gateway will fetch the key from the Vault, and then encrypt and store the object in a bucket. When a request is made to download the object, the Ceph Object Gateway will automatically retrieve the corresponding key from the Vault and decrypt the object. To upload an object, the Ceph object Gateway fetches the key from the Vault and then encrypts the object and stores it in the bucket. The Ceph Object Gateway retrieves the corresponding key from the Vault and decrypts the object when there is a request to download the object. Note The URL is constructed using the base address, set by the rgw_crypt_vault_addr option, and the path prefix, set by the rgw_crypt_vault_prefix option. Prerequisites A running Red Hat Ceph Storage cluster. Installation of the Ceph Object Gateway software. Installation of the HashiCorp Vault software. Access to a Ceph Object Gateway client node. Access to Amazon Web Services (AWS). Procedure Upload an object using the AWS command line client and provide the Secure Side Encryption (SSE) key ID in the request: For the Key/Value secret engine: Example (with SSE-KMS) Example (with SSE-S3) Note In the example, the Ceph Object Gateway would fetch the secret from http://vault-server:8200/v1/secret/data/myproject/mybucketkey For the Transit engine: Example (with SSE-KMS) Example (with SSE-S3) Note In the example, the Ceph Object Gateway would fetch the secret from http://vaultserver:8200/v1/transit/mybucketkey Additional Resources See the Install Vault documentation on Vault's project site for more information. 8.5. The Ceph Object Gateway and multi-factor authentication As a storage administrator, you can manage time-based one time password (TOTP) tokens for Ceph Object Gateway users. 8.5.1. Multi-factor authentication When a bucket is configured for object versioning, a developer can optionally configure the bucket to require multi-factor authentication (MFA) for delete requests. Using MFA, a time-based one time password (TOTP) token is passed as a key to the x-amz-mfa header. The tokens are generated with virtual MFA devices like Google Authenticator, or a hardware MFA device like those provided by Gemalto. Use radosgw-admin to assign time-based one time password tokens to a user. You must set a secret seed and a serial ID. You can also use radosgw-admin to list, remove, and resynchronize tokens. Important In a multi-site environment it is advisable to use different tokens for different zones, because, while MFA IDs are set on the user's metadata, the actual MFA one time password configuration resides on the local zone's OSDs. Table 8.1. Terminology Term Description TOTP Time-based One Time Password. Token serial A string that represents the ID of a TOTP token. Token seed The secret seed that is used to calculate the TOTP. It can be hexadecimal or base32. TOTP seconds The time resolution used for TOTP generation. TOTP window The number of TOTP tokens that are checked before and after the current token when validating tokens. TOTP pin The valid value of a TOTP token at a certain time. 8.5.2. Creating a seed for multi-factor authentication To set up multi-factor authentication (MFA), you must create a secret seed for use by the one-time password generator and the back-end MFA system. Prerequisites A Linux system. Access to the command line shell. Procedure Generate a 30 character seed from the urandom Linux device file and store it in the shell variable SEED : Example Print the seed by running echo on the SEED variable: Example Configure the one-time password generator and the back-end MFA system to use the same seed. Additional Resources For more information, see the solution Unable to create RGW MFA token for bucket . For more information, see The Ceph Object Gateway and multi-factor authentication . 8.5.3. Creating a new multi-factor authentication TOTP token Create a new multi-factor authentication (MFA) time-based one time password (TOTP) token. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. A secret seed for the one-time password generator and Ceph Object Gateway MFA was generated. Procedure Create a new MFA TOTP token: Syntax Set USERID to the user name to set up MFA on, set SERIAL to a string that represents the ID for the TOTP token, and set SEED to a hexadecimal or base32 value that is used to calculate the TOTP. The following settings are optional: Set the SEED_TYPE to hex or base32 , set TOTP_SECONDS to the timeout in seconds, or set TOTP_WINDOW to the number of TOTP tokens to check before and after the current token when validating tokens. Example Additional Resources For more information, see Creating a seed for multi-factor authentication . For more information, See Resynchronizing a multi-factor authentication token . 8.5.4. Test a multi-factor authentication TOTP token Test a multi-factor authentication (MFA) time-based one time password (TOTP) token. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Procedure Test the TOTP token PIN to verify that TOTP functions correctly: Syntax Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the latest PIN from the one-time password generator. Example If this is the first time you have tested the PIN, it may fail. If it fails, resynchronize the token. See Resynchronizing a multi-factor authentication token in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide . Additional Resources For more information, see Creating a seed for multi-factor authentication . For more information, see Resynchronizing a multi-factor authentication token . 8.5.5. Resynchronizing a multi-factor authentication TOTP token Resynchronize a multi-factor authentication (MFA) time-based one time password token. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Procedure Resynchronize a multi-factor authentication TOTP token in case of time skew or failed checks. This requires passing in two consecutive pins: the pin, and the current pin. Syntax Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, set PREVIOUS_PIN to the user's PIN, and set CURRENT_PIN to the user's current PIN. Example Verify the token was successfully resynchronized by testing a new PIN: Syntax Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the user's PIN. Example Additional Resources For more information, see Creating a new multi-factor authentication TOTP token . 8.5.6. Listing multi-factor authentication TOTP tokens List all multi-factor authentication (MFA) time-based one time password (TOTP) tokens that a particular user has. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Procedure List MFA TOTP tokens: Syntax Set USERID to the user name MFA is set up on. Example Additional Resources For more information, see Creating a new multi-factor authentication TOTP token . 8.5.7. Display a multi-factor authentication TOTP token Display a specific multi-factor authentication (MFA) time-based one time password (TOTP) token by specifying its serial. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Procedure Show the MFA TOTP token: Syntax Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token. Additional Resources For more information, see Creating a new multi-factor authentication TOTP token . 8.5.8. Deleting a multi-factor authentication TOTP token Delete a multi-factor authentication (MFA) time-based one time password (TOTP) token. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on a Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Procedure Delete an MFA TOTP token: Syntax Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token. Example Verify the MFA TOTP token was deleted: Syntax Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token. Example Additional Resources For more information, see The Ceph Object Gateway and multi-factor authentication . 8.5.9. Deleting an MFA-enabled versioned object Delete an MFA-enabled versioned object. Prerequisites A running Red Hat Ceph Storage cluster. Ceph Object Gateway is installed. You have root access on the Ceph Monitor node. An MFA TOTP token was created using radosgw-admin mfa create . Root user must be authenticated with an MFA-Object to delete a bucket that is versioning enabled. MFA-Object that you want to delete is in a bucket that is versioning enabled. Procedure Delete an MFA-enabled versioned object: Syntax Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token. Example Verify the MFA-enabled versioned object was deleted: Syntax

Commands:
[user@client ~]USD vi bucket-encryption.json
{ "Rules": [ { "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256" } } ] }
aws --endpoint-url=pass:q[_RADOSGW_ENDPOINT_URL_]:pass:q[_PORT_] s3api put-bucket-encryption --bucket pass:q[_BUCKET_NAME_] --server-side-encryption-configuration pass:q[_file://PATH_TO_BUCKET_ENCRYPTION_CONFIGURATION_FILE/BUCKET_ENCRYPTION_CONFIGURATION_FILE.json_]
[user@client ~]USD aws --endpoint-url=http://host01:80 s3api put-bucket-encryption --bucket testbucket --server-side-encryption-configuration file://bucket-encryption.json
aws --endpoint-url=pass:q[_RADOSGW_ENDPOINT_URL_]:pass:q[_PORT_] s3api get-bucket-encryption --bucket BUCKET_NAME
[user@client ~]USD aws --profile ceph --endpoint=http://host01:80 s3api get-bucket-encryption --bucket testbucket { "ServerSideEncryptionConfiguration": { "Rules": [ { "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256" } } ] } }
aws --endpoint-url= RADOSGW_ENDPOINT_URL : PORT s3api delete-bucket-encryption --bucket BUCKET_NAME
[user@client ~]USD aws --endpoint-url=http://host01:80 s3api delete-bucket-encryption --bucket testbucket
aws --endpoint-url= RADOSGW_ENDPOINT_URL : PORT s3api get-bucket-encryption --bucket BUCKET_NAME
[user@client ~]USD aws --endpoint=http://host01:80 s3api get-bucket-encryption --bucket testbucket An error occurred (ServerSideEncryptionConfigurationNotFoundError) when calling the GetBucketEncryption operation: The server side encryption configuration was not found
frontend http_web *:80 mode http default_backend rgw frontend rgw\u00ad-https bind *:443 ssl crt /etc/ssl/private/example.com.pem default_backend rgw backend rgw balance roundrobin mode http server rgw1 10.0.0.71:8080 check server rgw2 10.0.0.80:8080 check
frontend http_web *:80 mode http default_backend rgw frontend rgw\u00ad-https bind *:443 ssl crt /etc/ssl/private/example.com.pem http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https here we set the incoming HTTPS port on the load balancer (eg : 443) http-request set-header X-Forwarded-Port 443 default_backend rgw backend rgw balance roundrobin mode http server rgw1 10.0.0.71:8080 check server rgw2 10.0.0.80:8080 check
ceph config set client.rgw rgw_trust_forwarded_https true
systemctl enable haproxy systemctl start haproxy
ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=0
ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=1
ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=2
vault policy write rgw-kv-policy -<<EOF path "secret/data/*" { capabilities = ["read"] } EOF
vault policy write rgw-transit-policy -<<EOF path "transit/keys/*" { capabilities = [ "create", "update" ] denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] } } path "transit/keys/*" { capabilities = ["read", "delete"] } path "transit/keys/" { capabilities = ["list"] } path "transit/keys/+/rotate" { capabilities = [ "update" ] } path "transit/*" { capabilities = [ "update" ] } EOF
vault policy write old-rgw-transit-policy -<<EOF path "transit/export/encryption-key/*" { capabilities = ["read"] } EOF
ceph config set client.rgw rgw_crypt_s3_kms_backend vault
ceph config set client.rgw rgw_crypt_vault_auth agent ceph config set client.rgw rgw_crypt_vault_addr http:// VAULT_SERVER :8100
vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.role_id > PATH_TO_FILE
vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.secret_id > PATH_TO_FILE
pid_file = "/run/kv-vault-agent-pid" auto_auth { method "AppRole" { mount_path = "auth/approle" config = { role_id_file_path ="/root/vault_configs/kv-agent-role-id" secret_id_file_path ="/root/vault_configs/kv-agent-secret-id" remove_secret_id_file_after_reading ="false" } } } cache { use_auto_auth_token = true } listener "tcp" { address = "127.0.0.1:8100" tls_disable = true } vault { address = "http://10.8.128.9:8200" }
/usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
ceph config set client.rgw rgw_crypt_vault_secret_engine kv
ceph config set client.rgw rgw_crypt_vault_secret_engine transit
ceph config set client.rgw rgw_crypt_vault_namespace testnamespace1
ceph config set client.rgw rgw_crypt_vault_prefix /v1/secret/data
ceph config set client.rgw rgw_crypt_vault_prefix /v1/transit/export/encryption-key
http://vault-server:8200/v1/transit/export/encryption-key
systemctl restart ceph- CLUSTER_ID@SERVICE_TYPE . ID .service
systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service
ceph orch restart SERVICE_TYPE
ceph orch restart rgw
cephadm shell
ceph config set client.rgw rgw_crypt_sse_s3_backend vault
ceph config set client.rgw rgw_crypt_sse_s3_vault_auth agent ceph config set client.rgw rgw_crypt_sse_s3_vault_addr http:// VAULT_AGENT : VAULT_AGENT_PORT
ceph config set client.rgw rgw_crypt_sse_s3_vault_auth agent ceph config set client.rgw rgw_crypt_sse_s3_vault_addr http://vaultagent:8100
vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .rgw-ap-role-id > PATH_TO_FILE
vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .rgw-ap-secret-id > PATH_TO_FILE
pid_file = "/run/rgw-vault-agent-pid" auto_auth { method "AppRole" { mount_path = "auth/approle" config = { role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id" secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id" remove_secret_id_file_after_reading ="false" } } } cache { use_auto_auth_token = true } listener "tcp" { address = "127.0.0.1:8100" tls_disable = true } vault { address = "https://vaultserver:8200" }
/usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl
ceph config set client.rgw rgw_crypt_sse_s3_vault_secret_engine kv
ceph config set client.rgw rgw_crypt_sse_s3_vault_secret_engine transit
ceph config set client.rgw rgw_crypt_sse_s3_vault_namespace company/testnamespace1
ceph config set client.rgw rgw_crypt_sse_s3_vault_prefix /v1/secret/data
ceph config set client.rgw rgw_crypt_sse_s3_vault_prefix /v1/transit
http://vaultserver:8200/v1/transit
ceph config set client.rgw rgw_crypt_sse_s3_vault_verify_ssl true ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_cacert PATH_TO_CA_CERTIFICATE ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientcert PATH_TO_CLIENT_CERTIFICATE ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientkey PATH_TO_PRIVATE_KEY
ceph config set client.rgw rgw_crypt_sse_s3_vault_verify_ssl true ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_cacert /etc/ceph/vault.ca ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientcert /etc/ceph/vault.crt ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientkey /etc/ceph/vault.key
systemctl restart ceph- CLUSTER_ID@SERVICE_TYPE . ID .service
systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service
ceph orch restart SERVICE_TYPE
ceph orch restart rgw
vault secrets enable -path secret kv-v2
vault kv put secret/ PROJECT_NAME / BUCKET_NAME key=USD(openssl rand -base64 32)
vault kv put secret/myproject/mybucketkey key=USD(openssl rand -base64 32) ====== Metadata ====== Key Value --- ----- created_time 2020-02-21T17:01:09.095824999Z deletion_time n/a destroyed false version 1
vault secrets enable transit
vault write -f transit/keys/ BUCKET_NAME exportable=true
vault write -f transit/keys/mybucketkey exportable=true
vault write -f transit/keys/BUCKET_NAME/rotate exportable=true
vault write -f transit/keys/mybucketkey/rotate exportable=true
vault write -f transit/keys/BUCKET_NAME/config auto_rotate_period=DURATION
vault write -f transit/keys/mybucketkey/config auto_rotate_period=30d
vault read transit/export/encryption-key/BUCKET_NAME
vault read transit/export/encryption-key/mybucketkey
vault read transit/export/encryption-key/ BUCKET_NAME / VERSION_NUMBER
vault read transit/export/encryption-key/mybucketkey/1 Key Value --- ----- keys map[1:-gbTI9lNpqv/V/2lDcmH2Nq1xKn6FPDWarCmFM2aNsQ=] name mybucketkey type aes256-gcm96
[user@client ~]USD aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey
[user@client ~]USD aws s3api --endpoint http://rgw_host:8080 put-object --bucket my-bucket --key obj1 --body test_file_to_upload --server-side-encryption AES256
[user@client ~]USD aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey
[user@client ~]USD aws s3api --endpoint http://rgw_host:8080 put-object --bucket my-bucket --key obj1 --body test_file_to_upload --server-side-encryption AES256
[user@host01 ~]USD SEED=USD(head -10 /dev/urandom | sha512sum | cut -b 1-30)
[user@host01 ~]USD echo USDSEED 492dedb20cf51d1405ef6a1316017e
radosgw-admin mfa create --uid= USERID --totp-serial= SERIAL --totp-seed= SEED --totp-seed-type= SEED_TYPE --totp-seconds= TOTP_SECONDS --totp-window= TOTP_WINDOW
radosgw-admin mfa create --uid=johndoe --totp-serial=MFAtest --totp-seed=492dedb20cf51d1405ef6a1316017e
radosgw-admin mfa check --uid= USERID --totp-serial= SERIAL --totp-pin= PIN
radosgw-admin mfa check --uid=johndoe --totp-serial=MFAtest --totp-pin=870305 ok
radosgw-admin mfa resync --uid= USERID --totp-serial= SERIAL --totp-pin= PREVIOUS_PIN --totp=pin= CURRENT_PIN
radosgw-admin mfa resync --uid=johndoe --totp-serial=MFAtest --totp-pin=802021 --totp-pin=439996
radosgw-admin mfa check --uid= USERID --totp-serial= SERIAL --totp-pin= PIN
radosgw-admin mfa check --uid=johndoe --totp-serial=MFAtest --totp-pin=870305 ok
radosgw-admin mfa list --uid= USERID
radosgw-admin mfa list --uid=johndoe { "entries": [ { "type": 2, "id": "MFAtest", "seed": "492dedb20cf51d1405ef6a1316017e", "seed_type": "hex", "time_ofs": 0, "step_size": 30, "window": 2 } ] }
radosgw-admin mfa get --uid= USERID --totp-serial= SERIAL
radosgw-admin mfa remove --uid= USERID --totp-serial= SERIAL
radosgw-admin mfa remove --uid=johndoe --totp-serial=MFAtest
radosgw-admin mfa get --uid= USERID --totp-serial= SERIAL
radosgw-admin mfa get --uid=johndoe --totp-serial=MFAtest MFA serial id not found
radosgw-admin object rm --bucket <bucket_name> --object <object_name> --object-version <object_version> --totp-pin <totp_token>
[radosgw-admin object rm --bucket test-mfa --object obj1 --object-version 4SdKdSmpVAarLLdsFukjUVEwr-2oWfC --totp-pin 562813
radosgw-admin bucket list --bucket <bucket_name>