Chapter 3. Customizing the Block Storage service (cinder) Once the Block Storage service (cinder) is deployed, it can be further customized for your environment. Block Storage service customizations include: Quotas: you can define project-specific limits to constrain the use of Block Storage resources. Volume types: you can define the associated settings of each volume type. By default, all volume types are public and available to all projects (tenants) but you can create a private volume type to control or limit the access to it. Volume types allow you to provide different levels of volume performance for your users. You can include the following additional customization of your volume types: The back end that a volume uses: you can create a volume type for each supported back end. However, for volume types that do not specify a back end, the Block Storage scheduler selects the most appropriate back end. Configurable properties of the Block Storage back-end drivers: you can specify multiple back-end properties for the volume type by using key-value pairs called Extra Specs. Quality of Service (QoS) specifications: you can apply performance limits to volumes that users create by associating QoS specifications to each volume type. Volume encryption: you can create an encrypted volume type so that users can create encrypted volumes. Default volume types: you can specify which volume type is used when a cloud user creates a volume and does not specify a volume type. Some Block Storage features require the creation of an internal Block Storage project or tenant, that is also known as the service project, which is called cinder-internal . Manage or unmanage volumes and their snapshots: you can import volumes and their snapshots from other Block Storage volume services or remove volumes and their snapshots from this Block Storage volume service. Note All of these customizing procedures use the CLI which is faster, requires less setup, and provides more options than the Dashboard. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. To use Block Storage service cinder CLI commands, source the cloudrc file with the command USD source ./cloudrc before using them. If the cloudrc file does not exist, then you need to create it. For more information, see Creating the cloudrc file . 3.1. Viewing and modifying project quotas You can change or view the limits of the following Block Storage resource quotas for each project (tenant): volumes , the number of volumes allowed for each project. The default value is 10 . snapshots , the number of snapshots allowed for each project. The default value is 10 . gigabytes , the total amount of storage allowed for volumes for each project in gigabytes, this might also include the storage allowed for snapshots. The default values, also count the size of the snapshots against this limit of 1000 GB. Note The default value of the no_snapshot_gb_quota Block Storage initial parameter value includes the storage allocated for snapshots against the gigabytes quota . For more information, see Configuring initial Block Storage service defaults in Configuring persistent storage . per-volume-gigabytes , the maximum size of each volume in gigabytes. The default is -1 , which means that this quota is unspecified. You can obtain the usage of these Block Storage resource quotas for each project. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Note If the cloudrc file does not exist, then type in the exit command and create this file. For more information, see Creating the cloudrc file . Optional: List the projects to obtain the ID or name of the required project: View the current limits of the Block Storage resource quotas for a specific project: Replace <project> with the ID or name of the required project. This displays the limits of all the resource quotas for the specified project. Each volume type for this project, provides its own volumes, gigabytes, and, snapshots quotas that are unspecified by default, which means that they have a Limit of -1 . These 3 volume type quotas are suffixed by the volume type name. For instance the default volume type __DEFAULT__ , has the following associated quotas: volumes___DEFAULT__ , gigabytes___DEFAULT__ , and snapshots___DEFAULT__ . Optional: Modify the total size of all the Block Storage volumes, and the default value of the no_snapshot_gb_quota Block Storage initial parameter includes the size of the snapshots, that users can create for a project, in this amount: Replace <totalgb> with the total size, in gigabytes, of the Block Storage volumes, and if applicable the snapshots, that users can create for this project. Optional: Modify the maximum number of the Block Storage volumes that users can create for a project: Replace <maxvolumes> with the maximum number of volumes that users can create for this project. Optional: Modify the maximum size of the Block Storage volumes that users can create for a project: Replace <maxsize> with the maximum size, in gigabytes, of any Block Storage volume that users can create for this project. Optional: Modify the maximum number of the Block Storage volume snapshots that users can create for a project: Replace <maxsnapshots> with the maximum number of snapshots that users can create for this project. Optional: View the usage of these Block Storage resource quotas and, if necessary, review any changes to their limits for a specific project: Replace <project_id> with the ID of the project. Locate the relevant rows in this table that specify the Block Storage resource quotas for the specified project. This table also includes the quota defined for each volume type. Look at the following columns in this table: The In_use column indicates how much of each resource has been used. The Limit column indicates whether the quota limits have been adjusted from their default settings. In this example all the quota limits are adjusted. Exit the openstackclient pod: USD exit 3.2. Creating and configuring a volume type You can create volume types so that you can apply associated settings to each volume type. For instance, you can create volume types to provide different levels of performance for your cloud users. Create a volume type and add specific performance, resilience, and other Extra Specs as key-value pairs. Note By default, all volume types are public and accessible to all projects. If you need to create volume types with restricted access, then create a private volume type. For more information, see Creating and using private volume types . For smaller deployments that have a limited number of back ends, you can create volume types that target each back end by using the volume_backend_name property. In this case, volumes that use these volume types are scheduled on the specified back ends. For more information, see the example provided by Editing a volume type . For large deployments with large numbers of back ends this may not be a viable strategy. In this case it is better that the Block Storage scheduler determines the best back end for a volume, based on the configured scheduler filters. In addition to adding performance related properties, after creating a volume type, you can associate it with a Quality of Service specification that further configures volume performance. For more information, see Block Storage service (cinder) Quality of Service specifications . Prerequisites You must be a project administrator to create and configure volume types. Know the names of the keys and their required values for the back-end driver capabilities that you want to add to your volume types. For more information, see Listing back-end driver capabilities . Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create and configure a volume type. Add the required properties to the volume type, by specifying a separate --property <key>=<value> argument, as follows: Replace <key> with the property key. Important Ensure that you spell the <key> correctly. Any incorrectly spelled keys are added but will not work. Replace <value> with the required value of the <key> . Replace <volume_type_name> with the name of your volume type. For example: Exit the openstackclient pod: USD exit steps Associating a QoS specification with a volume type 3.2.1. Creating and using private volume types By default, all volume types are public and available to all projects (tenants). To control or to limit the access to a volume type, you can create a private volume type. Note By default, private volume types are only accessible to their creators. But all administrative users can view private volume types. Private volume types restrict access to volumes with certain attributes. Typically, these are settings that should only be usable by specific projects. For instance, new back ends or ultra-high performance configurations that are being tested. Prerequisites You must be a project administrator to create and configure volume types. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create a new private volume type. Replace <volume_type_name> with the name for the new private volume type. You can confirm that this is a private volume because the is_public field is set to False . For example: Optional: View private and public volume types. This provides a table of the volume types and the Is Public column indicates whether a volume type is private (False) or public (True). For example: This table also provides the name and ID of both public and private volume types, because you need the ID of the volume type to provide access to it. Optional: List the projects to obtain the ID of the required project: (Optional) Allow a project to access a private volume type: Replace <type_id> with the ID of the required private volume type. Replace <project_id> with the ID of the project that should access this private volume type. Note Access to a private volume type is granted at the project level. If you only know the name of a user of this project, then run the openstack user list command, which lists the name and tenant ID of all the configured users. (Optional) View which projects have access to a private volume type: The access_project_ids field of the resultant table provides the IDs of all the projects that can access this private volume type. (Optional) Remove project access from a private volume type: You can confirm this by running openstack volume type show <type_id> and ensuring this project is not specified in the access_project_ids field. Exit the openstackclient pod: USD exit 3.2.2. Listing back-end driver capabilities When creating volume types, the configurable properties or capabilities of the Block Storage back-end drivers are exposed and configured using key-value pairs called Extra Specs. Each back-end driver supports their own set of Extra Specs. For more information on the specific Extra Specs that a driver supports, see the back-end driver documentation. However, admins can always query the host of the Block Storage cinder-volume service directly to list the well-defined standard capabilities of its back-end driver. Prerequisites You must be a project administrator to query the Block Storage host directly. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Determine the host of the cinder-volume service: This command lists the properties of each Block Storage service ( cinder-backup , cinder-scheduler , and cinder-volume ). For example: The Host column specifies the host of each Block Storage service. However, the Host column of the cinder-volume service also provides the back end name by using this syntax: host@volume_back_end_name . Display the back-end driver capabilities of the Block Storage cinder-volume service: Replace <volsvchost> with the host of cinder-volume , provided in the table above. For example: The Key column provides the Extra Spec properties that you can set, while the Type column provides the required data type or valid values for these properties. Exit the openstackclient pod: USD exit 3.2.3. Editing a volume type You can edit a volume type by adding more properties to it or modifying the values of the existing properties. Note You can only edit a volume type if it is not in use. You can also delete volume types that you no longer need by using the openstack volume type delete and providing the name of the volume type. Prerequisites You must be a project administrator to edit volume types. Know the names of the keys and their required values for the back-end driver capabilities that you want to add to your volume types. For more information, see Listing back-end driver capabilities . Procedure Access the remote shell for the OpenStackClient pod from your workstation: Edit a volume type. You can use the openstack volume type list command to provide the list of configured volume types. Add or edit the properties of this volume type, by specifying a separate --property <key>=<value> argument, as follows: Replace <key> with the property key. Important Ensure that you spell the <key> correctly. Any incorrectly spelled keys are added but will not work. Replace <value> with the required value of the <key> . Replace <existing_volume_type_name> with the name of the required volume type. For example: If you wanted to configure a volume type to target a specific back end, you could add the following property to it: Note You can use the openstack volume backend pool list command to display a list of available back-end names, which uses this syntax: host@volume_backend_name#pool . Optional: This command does not provide any confirmation that the changes to the properties were successful. You can run the following command to review any changes made to this volume type: This command provides a table of the configuration details of this volume type and the properties field displays all the configured properties. For example: Exit the openstackclient pod: USD exit 3.3. Block Storage service (cinder) Quality of Service specifications You can apply performance limits to volumes that your cloud users create, by creating and associating Quality of Service (QoS) specifications to each volume type. For example, volumes that use higher performance QoS specifications could provide your users with more IOPS or users could assign lighter workloads to volumes that use lower performance QoS specifications to conserve resources. To create a QoS specification and associate it with a volume type, complete the following tasks: Create and configure the QoS specification . When you create a QoS specification you must choose the required consumer. The consumer determines where you want to apply the QoS limits and determines which QoS property keys are available to define the QoS limits. For more information about the available consumers, see Consumers of QoS specifications . You can create volume performance limits by setting the required QoS property keys to your deployment specific values. For more information on the QoS property keys provided by the Block Storage service (cinder), see Block Storage QoS property keys . Associating a QoS specification with a volume type . You can create, configure, and associate a QoS specification to a volume type by using the CLI. 3.3.1. Consumers of QoS specifications When you create a QoS specification you must choose the required consumer. The consumer determines where you want to apply the QoS limits and determines which QoS property keys are available to define the QoS limits. The Block Storage service (cinder) supports the following consumers of QoS specifications: front-end : The Compute service (nova) applies the QoS limits when the volume is attached to an instance. The Compute service supports all the QoS property keys provided by the Block Storage service. back-end : The back-end driver of the associated volume type applies the QoS limits. Each back-end driver supports their own set of QoS property keys. For more information on which QoS property keys the driver supports, see the back-end driver documentation. You would use the back-end consumer in cases where the front-end consumer is not supported. For instance, when attaching volumes to bare metal nodes through the Bare Metal Provisioning service (ironic). both : Both consumers apply the QoS limits, where possible. This consumer type therefore supports the following QoS property keys: When a volume is attached to an instance, then you can use every QoS property key that both the Compute service and the back-end driver supports. When the volume is not attached to an instance, then you can only use the QoS property keys that the back-end driver supports. 3.3.2. Block Storage QoS property keys The Block Storage service provides you with QoS property keys so that you can limit the performance of the volumes that your cloud users create. These limits use the following two industry standard measurements of storage volume performance: Input/output operations per second (IOPS) Data transfer rate, measured in bytes per second The consumer of the QoS specification determines which QoS property keys are supported. For more information, see Consumers of QoS specifications . Block Storage cannot perform error checking of QoS property keys, because some QoS property keys are defined externally by back-end drivers. Therefore, Block Storage ignores any invalid or unsupported QoS property key. Important Ensure that you spell the QoS property keys correctly. The volume performance limits that contain incorrectly spelled property keys are ignored. For both the IOPS and data transfer rate measurements, you can configure the following performance limits: Fixed limits Typically, fixed limits should define the average usage of the volume performance measurement. Burst limits Typically, burst limits should define periods of intense activity of the volume performance measurement. A burst limit makes allowance for an increased rate of activity for a specific time, while keeping the fixed limits low for average usage. Note The burst limits all use a burst length of 1 second. Total limits Specify a global limit for both the read and write operations of the required performance limit, by using the total_* QoS property key. Note Instead of using a total limit you can apply separate limits to the read and write operations or choose to limit only the read or write operations. Read limits Specify a limit that only applies to the read operations of the required performance limit, by using the read_* QoS property key. Note This limit is ignored when you specify a total limit. Write limits Specify a limit that only applies to the write operations of the required performance limit, by using the write_* QoS property key. Note This limit is ignored when you specify a total limit. You can use the following Block Storage QoS property keys to create volume performance limits for your deployment: Note The default value for all QoS property keys is 0 , which means that the limit is unrestricted. Table 3.1. Block Storage QoS property keys Performance limit Measurement unit QoS property keys Fixed IOPS IOPS total_iops_sec read_iops_sec write_iops_sec Fixed IOPS calculated by the size of the volume. For more information about the usage restrictions of these limits, see QoS limits that scale according to volume size . IOPS per GB total_iops_sec_per_gb read_iops_sec_per_gb write_iops_sec_per_gb Burst IOPS IOPS total_iops_sec_max read_iops_sec_max write_iops_sec_max Fixed data transfer rate Bytes per second total_bytes_sec read_bytes_sec write_bytes_sec Burst data transfer rate Bytes per second total_bytes_sec_max read_bytes_sec_max write_bytes_sec_max Size of an IO request when calculating IOPS limits. For more information, see Set the IO request size for IOPS limits . Bytes size_iops_sec 3.3.2.1. Set the IO request size for IOPS limits If you implement IOPS volume performance limits, you should also specify the typical IO request size to prevent users from circumventing these limits. If you do not then users could submit several large IO requests instead of a lot of smaller ones. Use the size_iops_sec QoS property key to specify the maximum size, in bytes, of a typical IO request. The Block Storage service uses this size to calculate the proportional number of typical IO requests for each IO request that is submitted, for example: size_iops_sec=4096 An 8 KB request is counted as two requests. A 6 KB request is counted as one and a half requests. Any request less than 4 KB is counted as one request. The Block Storage service only uses this IO request size limit when calculating IOPS limits. Note The default value of size_iops_sec is 0 , which ignores the size of IO requests when applying IOPS limits. 3.3.2.2. IOPS limits that scale according to volume size You can create IOPS volume performance limits that are determined by the capacity of the volumes that your users create. These Quality of Service (QoS) limits scale with the size of the provisioned volumes. For example, if the volume type has an IOPS limit of 500 per GB of volume size for read operations, then a provisioned 3 GB volume of this volume type would have a read IOPS limit of 1500. Important The size of the volume is determined when the volume is attached to an instance. Therefore if the size of the volume is changed while it is attached to an instance, these limits are only recalculated for the new volume size when this volume is detached and then reattached to an instance. You can use the following QoS property keys, specified in IOPS per GB, to create scalable volume performance limits: total_iops_sec_per_gb : Specify a global IOPS limit per GB of volume size for both the read and write operations. Note Instead of using a total limit you can apply separate limits to the read and write operations or choose to limit only the read or write operations. read_iops_sec_per_gb : Specify a IOPS limit per GB of volume size that only applies to the read operations. Note This limit is ignored when you specify a total limit. write_iops_sec_per_gb : Specify a IOPS limit per GB of volume size that only applies to the write operations. Note This limit is ignored when you specify a total limit. Important The consumer of the QoS specification containing these QoS limits can either be front-end or both , but not back-end . For more information, see Consumers of QoS specifications . 3.3.3. Creating and configuring a QoS specification A Quality of Service (QoS) specification is a list of volume performance QoS limits. You create each QoS limit by setting a QoS property key to your deployment specific value. To apply the QoS performance limits to a volume, you must associate the QoS specification with the required volume type. Prerequisites You must be a project administrator to create and configure QoS specifications. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create the QoS specification: Add the performance limits to the QoS specification, by specifying a separate --property <key>=<value> argument for each QoS limit, as follows: Replace <key> with the QoS property key of the required performance constraint. For more information, see Block Storage QoS property keys . Important Ensure that you spell the QoS property keys correctly. The volume performance limits that contain incorrectly spelled property keys are ignored. Replace <value> with your deployment-specific limit for this performance constraint, in the measurement unit required by the QoS property key. Optional: Replace <qos_spec_consumer> with the required consumer of this QoS specification. If not specified, the consumer defaults to both . For more information, see Consumers of QoS specifications . Replace <qos_spec_name> with the name of your QoS specification. Example: Exit the openstackclient pod: USD exit Edit an existing QoS specification Access the remote shell for the OpenStackClient pod from your workstation: Configure a created QoS specification to add other performance limits or to change the limits of the existing properties. You can add one or more --property <key>=<value> arguments. This command does not provide any confirmation that the changes to the properties were successful. You can run the following command to review any changes made to any QoS specification: This command provides a table of the configuration details of all the configured QoS specifications, the Properties column displays all the configured properties of each QoS specification. Exit the openstackclient pod: USD exit steps Associating a QoS specification with a volume type 3.3.4. Associating a QoS specification with a volume type You must associate a Quality of Service (QoS) specification with a volume type to apply the QoS limits to volumes. Important If a volume is already attached to an instance, then the QoS limits are only applied to this volume when the volume is detached and then reattached to this instance. Prerequisites The required volume type is created. For more information, see Creating and configuring a volume type . The required QoS specification is created. For more information, see Creating and configuring a QoS specification . Procedure Access the remote shell for the OpenStackClient pod from your workstation: Associate the required QoS specification with the required volume type: Replace <qos_spec_name> with the name or ID of the QoS specification. You can run the openstack volume qos list command to list the name and ID of all the QoS specifications. Replace <volume_type> with the name or ID of the volume type. You can run the openstack volume type list command to list the name and ID of all the volume types. Verify that the QoS specification has been associated: The Associations column of the output table provides the names of the volume types that are associated with each QoS specification. For example: Exit the openstackclient pod: USD exit 3.3.5. Disassociating a QoS specification from volume types You can disassociate a Quality of Service (QoS) specification from a volume type when you no longer want the QoS limits to be applied to volumes of that volume type. You can either disassociate a specific volume type, or all the volumes types when more than one volume type is associated to the same QoS specification. Important If a volume is already attached to an instance, then the QoS limits are only removed from this volume when the volume is detached and then reattached to this instance. Prerequisites You must be a project administrator to create, configure, associate, and disassociate QoS specifications. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Disassociate the volume types associated with the QoS specification: To disassociate a specific volume type associated with the QoS specification: Replace <qos_spec_name> with the name or ID of the QoS specification. You can run the openstack volume qos list command to list the name and ID of all the QoS specifications. Replace <volume_type> with the name or ID of the volume type associated with this QoS specification. You can run the cinder type-list command to list the name and ID of all the volume types. To disassociate all volume types associated with the QoS specification: Verify that the QoS specification has been disassociated: The Associations column of this QoS specification should either not contain the specified volume type or should be empty when all the volume types were disassociated. Exit the openstackclient pod: USD exit 3.4. Configuring Block Storage volume encryption To create encrypted volumes, you need an encrypted volume type that specifies the encryption provider, cipher, key size, and control location. Prerequisites You must be a project administrator to create encrypted volume types. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create an encrypted volume type: Replace <provider_name> with the luks encryption provider. Replace <cipher_name> with the aes-xts-plain64 encryption cipher. Replace <key_size> with the 256 encryption key size. Replace <control_location> with front-end , so the Compute service (nova) performs the encryption when the volume is attached a Compute node. Replace <encrypted_volume_type_name> with the name to use for the encrypted volume type. For example: The resultant table for an encrypted volume type provides the additional encryption field, which provides the encryption_id . In this example the encryption field has the following value: cipher='aes-xts-plain64', control_location='front-end', encryption_id='039c8b02-2e15-41fa-899d-6e2a555b7fc8', key_size='256', provider='luks' Exit the openstackclient pod: USD exit steps Create an encrypted volume from your encrypted volume type. 3.5. Configuring Block Storage volumes that can be attached to multiple instances at the same time To attach a Block Storage (cinder) volume to multiple instances at the same time, you must create this volume from or retype this volume to a multi-attach volume type. These multiple instances can then simultaneously read data from and write data to this volume. Warning You must use a multi-attach or cluster-aware file system to manage write operations from multiple instances. Failure to do so causes data corruption. For example, you can create a multi-attach volume on a NFS back end and then attach this volume to multiple instances that can use this volume as shared storage. Limitations of multi-attach volumes Your Block Storage (cinder) back-end driver must support multi-attach volumes. The Ceph RBD driver is supported. Contact Red Hat support to verify that the multiattach volume property is supported by your vendor plugin. For more information about the certification of your vendor plugin, see https://catalog.redhat.com/search?searchType=software&target_platforms=Red%20Hat%20OpenStack%20Platform&p=1&subcategories=Storage . When you attach a multi-attach volume, some hypervisors require special considerations, such as when you disable caching. Read-only multi-attach volumes are not supported. Live migration of multi-attach volumes is not available. Encryption of multi-attach volumes is not supported. Multi-attach volumes are not supported by the Bare Metal Provisioning service (ironic) virt driver. Multi-attach volumes are supported by only the libvirt virt driver. When a volume is not in use and its status is available , you can retype this volume to be multi-attach capable or retype a multi-attach capable volume to make it incapable of attaching to multiple instances. You cannot retype an attached volume from a multi-attach type to a non-multi-attach type, or from a non-multi-attach type to a multi-attach type. You cannot use multi-attach volumes that have multiple read write attachments as the source or destination volume during an attached volume migration. You cannot attach multi-attach volumes to shelved offloaded instances. Simultaneous API calls, such as create or resize, to instances attached to the same multi-attach volume can fail. For more information, see https://access.redhat.com/solutions/7077470 . Prerequisites You must be a project administrator to create multi-attach volume types. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create a multi-attach volume type: Replace <multi-attach_volume_type_name> with the name of your multi-attach volume type. Exit the openstackclient pod: USD exit steps Create a volume from your multi-attach volume type and attach this volume to multiple instances. 3.6. Configuring project-specific default volume types If you create a volume and do not specify a volume type, the Block Storage service (cinder) uses the default volume type. You can use the default_volume_type parameter, when initially configuring the Block Storage service, to define the general default volume type that applies to all projects. By default, this volume type is called __DEFAULT__ . For more information, see Configuring initial Block Storage service defaults in Configuring persistent storage . If your deployment uses project-specific volume types, ensure that you define default volume types for each project. In this case, Block Storage uses the project-specific volume type instead of the general default volume type. The following deployment types need project-specific default volume types: A distributed deployment spanning many availability zones (AZs). Each AZ is in its own project and has its own volume types. A multi-department, corporate deployment. Each department is in its own project and has its own specialized volume type. Prerequisites At least one volume type in each project that will be the project-specific default volume type. For more information, see Creating and configuring a volume type . Block Storage REST API microversion 3.62 or later. Only project administrators can define, clear, or list default volume types for their projects. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Note If the cloudrc file does not exist, then type in the exit command and create this file. For more information, see Creating the cloudrc file . Configure the default volume type for a project: Replace <volume_type> with the name or ID of the required volume type. To find the volume type name and ID, use the volume type list command. Replace <project_id> with the ID of the appropriate project. To find the project ID, use the openstack project list command. (Optional) Remove a default volume type for a project: Replace <project_id> with the ID of the appropriate project. To find the project ID, use the openstack project list command. (Optional) List the default volume type for a project: Replace <project_id> with the ID of the appropriate project. To find the project ID, use the openstack project list command. Exit the openstackclient pod: USD exit 3.7. Creating and configuring an internal project for the Block Storage service (cinder) Some Block Storage features require an internal Block Storage project or tenant, that is also known as the service project, which is called cinder-internal . The Block Storage service uses this project to manage resources that this service creates to ensure that these resources are not counted towards the users' project quota limits. For example: the volumes that are created when images are cached for frequent volume cloning or temporary copies of volumes being migrated. Since this cinder-internal project is also subject to the default project quotas, you must adjust the volumes quota for this project, which is limited to only 10 volumes, by default. You can either set the limit of the volumes quota to be unlimited (-1) or provide a number, to ensure the effective use of your available volume storage, such as 50. You might also need to change the gigabytes quota that limits the maximum size of the created volumes for this project, in gigabytes, which is set to 1000 by default. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Create a generic project and user, both named cinder-internal , as follows: Modify the maximum number of volumes that the Block Storage service can create for this cinder-internal project: Replace <maxnum> with the maximum number of volumes that the Block Storage service can create for this cinder-internal project. Optional: Modify the maximum total size of all the volumes that the Block Storage service can create for this cinder-internal project: Replace <maxgb> with the maximum total size, in gigabytes, of the volumes that the Block Storage service can create for this cinder-internal project. Exit the openstackclient pod: USD exit 3.8. Migrating a volume between back ends You can migrate volumes between back ends within, and across, availability zones (AZs). In highly customized deployments or in situations in which you must retire a storage system, an administrator can migrate volumes. In both use cases, multiple storage systems either share the same volume_backend_name property, or this property is undefined. Restrictions for migrated volumes The volume cannot be replicated. The destination back end must be different from the current back end of the volume. The existing volume type must be valid for the new back end, which means that the following must be true: The volume type must either not have the volume_backend_name defined, or both Block Storage back ends must be configured with the same volume_backend_name . Both back ends must support the same features configured in the volume type, such as support for thin provisioning, support for thick provisioning, or other feature configurations. Note Moving volumes from one back end to another might require extensive time and resources. For more information, see Restrictions and performance constraints when moving volumes . Prerequisites You must be a project administrator to migrate volumes. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Retrieve the name of the destination back end. This lists the destination back-end names, which use this syntax: host@volume_backend_name#pool . For example: Migrate a volume from one back end to another. Replace <volume> with the volume ID or name of the originating back end. Replace <new_back_end_host> with the host of the destination back end. Exit the openstackclient pod: USD exit 3.8.1. Restrictions and performance constraints when moving volumes Red Hat supports moving volumes between back ends within and across availability zones (AZs), with the following restrictions: Volumes must have either have a status of available or in-use to be moved. Support for in-use volumes is driver dependent. Volumes cannot have snapshots. Volumes cannot belong to a group. Moving available volumes You can move available volumes between all back ends, but performance depends on the back ends that you use. Many back ends support assisted migration. For more information about back-end support for assisted migration, contact the vendor. With assisted migration, the back end optimizes the movement of the data from the source back end to the destination back end, but both back ends must be from the same vendor. Note Red Hat supports back-end assisted migrations only with multi-pool back ends or when you use the cinder migrate operation for single-pool back ends, such as RBD. When assisted migrations between back ends are not possible, the Block Storage service performs a generic volume migration. Generic volume migration requires volumes on both back ends to be connected before the Block Storage (cinder) service moves data from the source volume to the Controller node and from the Controller node to the destination volume. The Block Storage service seamlessly performs the process regardless of the type of storage from the source and destination back ends. Important Ensure that you have adequate bandwidth before you perform a generic volume migration. The duration of a generic volume migration is directly proportional to the size of the volume, which makes the operation slower than assisted migration. Moving in-use volumes In-use multi-attach volumes cannot be moved while they are attached to more than one nova instance. Non block devices are not supported, which limits storage protocols on the target back end to be iSCSI, Fibre Channel (FC), and RBD. There is no optimized or assisted option for moving in-use volumes. When you move in-use volumes, the Compute service (nova) must use the hypervisor to transfer data from a volume in the source back end to a volume in the destination back end. This requires coordination with the hypervisor that runs the instance where the volume is in use. The Block Storage service (cinder) and the Compute service work together to perform this operation. The Compute service manages most of the work, because the data is copied from one volume to another through the Compute node. Important Ensure that you have adequate bandwidth before you move in-use volumes. The duration of this operation is directly proportional to the size of the volume, which makes the operation slower than assisted migration. 3.9. Manage or unmanage volumes and their snapshots You can add volumes to or remove volumes from the Block Storage volume service ( cinder-volume ) by using the cinder manage and cinder unmanage commands. Important You cannot manage or unmanage encrypted volumes. Typically the Block Storage volume service manages the volumes that it creates, so that it can, for instance, list, attach, and delete these volumes. You can use the cinder unmanage command to remove a volume from the Block Storage volume service, so that it will no longer list, attach or delete this volume. Note You cannot unmanage a volume if it has snapshots. In this case, you must unmanage all of the snapshots before you unmanage a volume, by using the cinder snapshot-unmanage command. You can use the cinder manage command to add a volume to the Block Storage volume service, so that it can, for instance, list, attach, and delete this volume. Then you can add the snapshots of this volume, by using the cinder snapshot-manage command. You can use the cinder manageable-list command to determine whether there are volumes in the storage arrays of the Block Storage volume service that are not being managed. The volumes in this list are typically volumes that users have unmanaged or which have been created manually on a storage array without using the Block Storage volume service. Similarly, the cinder snapshot-manageable-list command lists all the manageable snapshots. The syntax of the cinder manage`and `cinder snapshot-manage commands is back end specific, because the properties required to identify the volume are back end specific. Most back ends support either or both of the source-name and source-id properties, others require additional properties to be set. Some back ends can list which volumes are manageable and what parameters need to be passed. For those back ends that do not, please refer to the vendor documentation. The syntax of the cinder unmanage and cinder snapshot-unmanage commands is not back end specific, you must specify the required volume name or volume ID. Usage scenarios You can use these Block Storage commands when upgrading your Red Hat OpenStack Services on OpenShift (RHOSO) deployment in parallel, by keeping your existing RHOSO version running while you deploy the new version of RHOSO. In this scenario, you must unmanage all of the snapshots and then unmanage the volume to remove a volume from your existing RHOSO and then you must manage this volume and all of its snapshots to add this volume and its snapshots to the new version of RHOSO. In this way, you can move your volumes and their snaphots to your new RHOSO version while running your existing cloud. Another possible scenario is if you had a bare metal machine using a volume in one of your storage arrays. Then you decide to move the software running on this machine into the cloud but you still want to use this volume. In this scenario, you use the cinder manage command to add this volume to the Block Storage volume service. 3.10. Creating the cloudrc file The Block Storage service (cinder) uses both openstack and cinder client commands, within the openstackclient pod. When cinder client commands are required, then the cloudrc file must be created on the openstackclient pod to enable their usage. Once created, the cloudrc file persists for the lifetime of the openstackclient pod. If the cloudrc does not exist, for instance it cannot be sourced, then use the following command to create it:

Commands:
oc rsh -n openstack openstackclient source ./cloudrc
openstack project list
openstack quota show <project>
openstack quota show c2c1da89ed1648fc8b4f35a045f8d34c +-----------------------+-------+ | Resource | Limit | +-----------------------+-------+ | volumes | 10 | | snapshots | 10 | | gigabytes | 1000 | | volumes___DEFAULT__ | -1 | | gigabytes___DEFAULT__ | -1 | | snapshots___DEFAULT__ | -1 | | per-volume-gigabytes | -1 | +-----------------------+-------+
openstack quota set --gigabytes <totalgb> <project>
openstack quota set --volumes <maxvolumes> <project>
openstack quota set --per-volume-gigabytes <maxsize> <project>
openstack quota set --snapshots <maxsnapshots> <project>
cinder quota-usage <project_id>
cinder quota-usage c2c1da89ed1648fc8b4f35a045f8d34c +-----------------------+--------+----------+-------+-----------+ | Type | In_use | Reserved | Limit | Allocated | +-----------------------+--------+----------+-------+-----------+ | gigabytes | 750 | 0 | 1500 | | | gigabytes___DEFAULT__ | 0 | 0 | -1 | | | groups | 0 | 0 | 10 | | | per_volume_gigabytes | 0 | 0 | 300 | | | snapshots | 1 | 0 | 7 | | | snapshots___DEFAULT__ | 0 | 0 | -1 | | | volumes | 5 | 0 | 15 | | | volumes___DEFAULT__ | 0 | 0 | -1 | | +-----------------------+--------+----------+-------+-----------+
exit
oc rsh -n openstack openstackclient
openstack volume type create --property <key>=<value> <volume_type_name>
openstack volume type create --property thin_provisioning=true --property compression=false MyVolumeType +-------------+-----------------------------------------------+ | Field | Value | +-------------+-----------------------------------------------+ | description | None | | id | c244205c-fb22-4076-9780-edebe55889bc | | is_public | True | | name | MyVolumeType | | properties | compression='false', thin_provisioning='true' | +-------------+-----------------------------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume type create --private <volume_type_name>
openstack volume type create --private MyVolType2 +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | description | None | | id | 20659377-935d-4253-8b82-ae12c4710288 | | is_public | False | | name | MyVolType2 | +-------------+--------------------------------------+
openstack volume type list
+--------------------------------------+-------------+-----------+ | ID | Name | Is Public | +--------------------------------------+-------------+-----------+ | 271f9b90-8186-4143-aaed-09e91820d852 | MyVolType3 | True | | 20659377-935d-4253-8b82-ae12c4710288 | MyVolType2 | False | | 28b5ca7f-0eb0-43ce-bf0a-898fab92d43b | MyVolType1 | True | | 0875116b-27d6-493c-aca6-f62de4d58614 | __DEFAULT__ | True | +--------------------------------------+-------------+-----------+
openstack project list
openstack volume type set <type_id> --project <project_id>
openstack volume type show <type_id>
openstack volume type unset <type_id> --project <project_id>
exit
oc rsh -n openstack openstackclient
openstack volume service list
+------------------+---------------------------+------+--------- | Binary | Host | Zone | Status +------------------+---------------------------+------+--------- | cinder-scheduler | cinder-scheduler-0 | nova | enabled | cinder-backup | cinder-backup-0 | nova | enabled | cinder-volume | cinder-volume-lvm-iscsi-0@lvm | nova | enabled +------------------+---------------------------+------+---------
openstack volume backend capability show <volsvchost>
openstack volume backend capability show cinder-volume-lvm-iscsi-0 +-------------------+---------------------+---------+-------------------------+ | Title | Key | Type | Description | +-------------------+---------------------+---------+-------------------------+ | Thin Provisioning | thin_provisioning | boolean | Sets thin provisioning. | | Compression | compression | boolean | Enables compression. | | QoS | qos | boolean | Enables QoS. | | Replication | replication_enabled | boolean | Enables replication. | +-------------------+---------------------+---------+-------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume type set --property <key>=<value> <existing_volume_type_name>
openstack volume type set --property volume_backend_name=lvm MyVolumeType
openstack volume type show <volume_type_name>
openstack volume type show MyVolumeType +--------------------+--------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------+ | access_project_ids | None | | description | None | | id | c244205c-fb22-4076-9780-edebe55889bc | | is_public | True | | name | MyVolumeType | | properties | compression='false', thin_provisioning='true', volume_backend_name='lvm' | | qos_specs_id | None | +--------------------+--------------------------------------------------------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume qos create [--consumer <qos_spec_consumer>] --property <key>=<value> <qos_spec_name>
openstack volume qos create --property read_iops_sec=5000 --property write_iops_sec=7000 --consumer front-end myqoslimits +------------+---------------------------------------------+ | Field | Value | +------------+---------------------------------------------+ | consumer | front-end | | id | 9fc9a481-28e9-49b8-84eb-f0a476cc89a5 | | name | myqoslimits | | properties | read_iops_sec='5000', write_iops_sec='7000' | +------------+---------------------------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume qos set --property <key>=<value> <qos_spec_name>
openstack volume qos list
exit
oc rsh -n openstack openstackclient
openstack volume qos associate <qos_spec_name> <volume_type>
openstack volume qos list
+--------------------------------------+--------------+-----------+--------------+-------------------------------------------------------------------+ | ID | Name | Consumer | Associations | Properties | +--------------------------------------+--------------+-----------+--------------+-------------------------------------------------------------------+ | 9fc9a481-28e9-49b8-84eb-f0a476cc89a5 | myqoslimits | both | MyVolType | read_iops_sec='6500', size_iops_sec='4096', write_iops_sec='7500' | +--------------------------------------+--------------+-----------+--------------+-------------------------------------------------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume qos disassociate <qos_spec_name> --volume-type <volume_type>
openstack volume qos disassociate <qos_spec_name> --all
openstack volume qos list
exit
oc rsh -n openstack openstackclient
openstack volume type create --encryption-provider <provider_name> --encryption-cipher <cipher_name> --encryption-key-size <key_size> --encryption-control-location <control_location> <encrypted_volume_type_name>
openstack volume type create --encryption-provider luks --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end MyEncryptedVolType
exit
oc rsh -n openstack openstackclient
openstack volume type create --property multiattach="<is> True" <multi-attach_volume_type_name>
exit
oc rsh -n openstack openstackclient source ./cloudrc
cinder --os-volume-api-version 3.62 default-type-set <volume_type> <project_id>
cinder --os-volume-api-version 3.62 default-type-unset <project_id>
cinder --os-volume-api-version 3.62 default-type-list --project <project_id>
exit
oc rsh -n openstack openstackclient
openstack project create --enable --description "Block Storage Internal Project" cinder-internal +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | Block Storage Internal Project | | domain_id | default | | enabled | True | | id | 670615550a5d4126b22953f76f380397 | | is_domain | False | | name | cinder-internal | | options | {} | | parent_id | default | | tags | [] | +-------------+----------------------------------+ openstack user create --project cinder-internal cinder-internal +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | default_project_id | 670615550a5d4126b22953f76f380397 | | domain_id | default | | enabled | True | | id | a6b3576f345346a590f2c8292f5cbd60 | | name | cinder-internal | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+
openstack quota set --volumes <maxnum> cinder-internal
openstack quota set --gigabytes <maxgb> cinder-internal
exit
oc rsh -n openstack openstackclient
openstack volume backend pool list
+--------------------------------+ | Name | +--------------------------------+ | cinder-volume-lvm-0@lvm#lvm | | cinder-volume-lvm2-0@lvm2#lvm2 | +--------------------------------+
openstack volume migrate --host <new_back_end_host> <volume>
exit
oc rsh openstackclient bash -c ' cat .config/openstack/* | while read key val; do case USDkey in "auth_url:") var=OS_AUTH_URL ;; "username:") var=OS_USERNAME ;; "password:") var=OS_PASSWORD ;; "project_name:") var=OS_PROJECT_NAME ;; "project_domain_name:") var=OS_PROJECT_DOMAIN_NAME ;; "user_domain_name:") var=OS_USER_DOMAIN_NAME ;; *) var="" ;; esac [ -z "USDvar" ] || echo "export USD{var}=USD{val}" done > cloudrc '