Chapter 4. Configuring the Block Storage service (cinder) The Block Storage service (cinder) provides access to remote block storage devices through volumes to provide persistent storage. The Block Storage service has three mandatory services; api , scheduler , and volume ; and one optional service, backup . All Block Storage services use the cinder section of the OpenStackControlPlane custom resource (CR) for their configuration: Global configuration options are applied directly under the cinder and template sections. Service specific configuration options appear under their associated sections. The following example demonstrates all of the sections where Block Storage service configuration is applied and what type of configuration is applied in each section: 4.1. Terminology The following terms are important to understanding the Block Storage service (cinder): Storage back end: A physical storage system where volume data is stored. Cinder driver: The part of the Block Storage service that enables communication with the storage back end.It is configured with the volume_driver and backup_driver options. Cinder back end: A logical representation of the grouping of a cinder driver with its configuration. This grouping is used to manage and address the volumes present in a specific storage back end. The name of this logical construct is configured with the volume_backend_name option. Storage pool: A logical grouping of volumes in a given storage back end. Cinder pool: A representation in the Block Storage service of a storage pool. Volume host: The way the Block Storage service address volumes. There are two different representations, short ( <hostname>@<backend-name> ) and full ( <hostname>@<backend-name>#<pool-name> ). Quota: Limits defined per project to constrain the use of Block Storage specific resources. 4.2. Block Storage service (cinder) enhancements in Red Hat OpenStack Services on OpenShift (RHOSO) The following functionality enhancements have been integrated into the Block Storage service: Ease of deployment for multiple volume back ends. Back end deployment does not affect running volume back ends. Back end addition and removal does not affect running back ends. Back end configuration changes do not affect other running back ends. Each back end can use its own vendor-specific container image. It is no longer necessary to build a custom image that holds dependencies from two drivers. Pacemaker has been replaced by Red Hat OpenShift Container Platform (RHOCP) functionality. Improved methods for troubleshooting the service code. 4.3. Configuring transport protocols Deployments use different transport protocols to connect to volumes. The Block Storage service (cinder) supports the following transport protocols: iSCSI Fibre Channel (FC) NVMe over TCP (NVMe-TCP) NFS Red Hat Ceph Storage RBD Control plane services that use volumes, such as the Block Storage service (cinder) volume and backup services, may require the support of the Red Hat OpenShift Container Platform (RHOCP) cluster to use iscsid and multipathd modules, depending on the storage array in use. These modules must be available on all nodes where these volume-dependent services execute. To use these transport protocols, a MachineConfig CR is created to define where these modules execute. For more information on a MachineConfig , see Understanding the Machine Config operator . Important Using a MachineConfig to change the configuration of a node causes the node to reboot. Consult with your RHOCP administrator before applying a 'MachineConfig` to ensure the integrity of RHOCP workloads. The procedures in this section are meant as a guide to the general configuration of these protocols. Storage back end vendors will supply configuration information on how to connect to their specific solution. In addition to protocol specific configuration, configure multipathing regardless of the transport protocol used. After you have completed the transport protocol configuration, see Configuring multipathing for the procedure. Note These services are automatically started on EDPM nodes. 4.3.1. Configuring the iSCSI protocol Connecting to iSCSI volumes from the RHOCP nodes requires the iSCSI initiator service. There must be a single instance of the iscsid service module for the normal RHOCP usage, OpenShift CSI plugins usage, and the RHOSO services. Apply a MachineConfig to the applicable nodes to configure nodes to use the iSCSI protocol. Note If the iscsid service module is already running, this procedure is not required. Procedure Create a MachineConfig CR to configure the nodes for the iscsid module. The following example starts the iscsid service with a default configuration in all RHOCP worker nodes: Save the file. Apply the MachineConfig CR file. Replace <machine_config_file> with the name of your MachineConfig CR file. 4.3.2. Configuring the Fibre Channel protocol There is no additional node configuration required to use the Fibre Channel protocol to connect to volumes. It is mandatory though that all nodes using Fibre Channel have an Host Bus Adapter (HBA) card. Unless all worker nodes in your RHOCP deployment have an HBA card, you must use a nodeSelector in your control plane configuration to select which nodes are used for volume and backup services, as well as the Image service instances that use the Block Storage service for their storage back end. 4.3.3. Configuring the NVMe over TCP (NVMe-TCP) protocol Connecting to NVMe-TCP volumes from the RHOCP nodes requires the nvme kernel modules. Procedure Create a MachineConfig CR to configure the nodes for the nvme kernel modules. The following example starts the nvme kernel modules with a default configuration in all RHOCP worker nodes: Save the file. Apply the MachineConfig CR file. Replace <machine_config_file> with the name of your MachineConfig CR file. After the nodes have rebooted, verify the nvme-fabrics module are loaded and support ANA on a host: Note Even though ANA does not use the Linux Multipathing Device Mapper, multipathd must be running for the Compute nodes to be able to use multipathing when connecting volumes to instances. 4.3.4. Configuring multipathing Configuring multipathing on RHOCP nodes requires a MachineConfig CR that creates a multipath.conf file on a node and starts the service. Note The example provided in this procedure creates only a minimal multipath.conf file. Production deployments may require hardware vendor specific additions as appropriate to your environment. Consult with the appropriate systems administrators for any values required for your deployment. Procedure Create a MachineConfig CR to configure the nodes multipathing. The following example creates a multipath.conf file and starts the multipathd module on all RHOCP nodes: Note The following would be the contents of the multipath.conf created by this example: Save the file. Apply the MachineConfig CR file. Replace <machine_config_file> with the name of your MachineConfig CR file. Note In RHOSO deployments, the use_multipath_for_image_xfer configuration option is enabled by default. 4.4. Configuring initial defaults The Block Storage service (cinder) has a set of initial defaults that should be configured when the service is first enabled. They must be defined in the main customServiceConfig section. Once deployed, these initial defaults are modified using the openstack client. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the Block Storage service global configuration. The following example demonstrates a Block Storage service initial configuration: For a complete list of all initial default parameters, see Initial default parameters . Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.4.1. Initial default parameters These initial default parameters should be configured when the service is first enabled. Parameter Description default_volume_type Provides the default volume type for all users. The default type of any non-default value will not be automatically created. The default value is __DEFAULT__ . no_snapshot_gb_quota Determines if the size of snapshots count against the gigabyte quota in addition to the size of volumes. The default is false , which means that the size of the snapshots are included in the gigabyle quota. per_volume_size_limit Provides the maximum size of each volume in gigabytes. The default is -1 (unlimited). quota_volumes Provides the number of volumes allowed for each project. The default value is 10 . quota_snapshots Provides the number of snapshots allowed for each project. The default value is 10 . quota_groups Provides the number of volume groups allowed for each project, which includes the consistency groups. The default value is 10 . quota_gigabytes Provides the total amount of storage for each project, in gigabytes, allowed for volumes, and depending upon the configuration of the no_snapshot_gb_quota initial parameter this might also include the size of the snapshots. The default values, also count the size of the snapshots against this limit of 1000 GB. quota_backups Provides the number backups allowed for each project. The default value is 10 . quota_backup_gigabytes Provides the total amount of storage for each project, in gigabytes, allowed for backups. The default is 1000 . 4.5. Configuring the API service The Block Storage service (cinder) provides an API interface for all external interaction with the service for both users and other RHOSO services. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the configuration for the internal Red Hat OpenShift Container Platform (RHOCP) load balancer. The following example demonstrates a load balancer configuration: Edit the CR file and add the configuration for the number of API service replicas. Run the cinderAPI service in an Active-Active configuration with three replicas. The following example demonstrates configuring the cinderAPI service to use three replicas: Edit the CR file and configure cinderAPI options. These options are configured in the customServiceConfig section under the cinderAPI section. The following example demonstrates configuring cinderAPI service options and enabling debugging on all services: For a listing of commonly used cinderAPI service option parameters, see API service option parameters . Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.5.1. API service option parameters API service option parameters are provided for the configuration of the cinderAPI portions of the Block Storage service. Parameter Description api_rate_limit Provides a value to determine if the API rate limit is enabled. The default is false . debug Provides a value to determine whether the logging level is set to DEBUG instead of the default of INFO . The default is false . The logging level can be dynamically set without restarting. osapi_max_limit Provides a value for the maximum number of items a collection resource returns in a single response. The default is 1000 . osapi_volume_workers Provides a value for the number of workers assigned to the API component. The default is the number of CPUs available. 4.6. Configuring the scheduler service The Block Storage service (cinder) has a scheduler service ( cinderScheduler ) that is responsible for making decisions such as selecting which back end receives new volumes, whether there is enough free space to perform an operation or not, and deciding where an existing volume should be moved to on some specific operations. Use only a single instance of cinderScheduler for scheduling consistency and ease of troubleshooting. While cinderScheduler can be run with multiple instances, the service default replicas: 1 is the best practice. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the configuration for the service down detection timeouts. The following example demonstrates this configuration: 1 The number of seconds between Block Storage service components reporting an operational state in the form of a heartbeat through the database. The default is 10 . 2 The maximum number of seconds since the last heartbeat from the component for it to be considered non-operational. The default is 60 . Note Configure these values at the cinder level of the CR instead of the cinderScheduler so that these values are applied to all components consistently. Edit the CR file and add the configuration for the statistics reporting interval. The following example demonstrates configuring these values at the cinder level to apply them globally to all services: 1 The number of seconds between requests from the volume for usage statistics from the back end. The default is 60 2 The number of seconds between requests from the volume for usage statistics from backup service. The default is 60 . The following example demonstrates configuring these values at the cinderVolume and cinderBackup level to customize settings at the service level. 1 The number of seconds between requests from the volume for usage statistics from the back end. The default is 60 2 The number of seconds between requests from the volume for usage statistics from backup service. The default is 60 . Note The generation of usage statistics can be resource intensive for some back ends. Setting these values too low can affect back end performance. You may need to tune the configuration of these settings to better suit individual back ends. Perform any additional configuration necessary to customize the cinderScheduler service. For more configuration options for the customization of the cinderScheduler service, see Scheduler service parameters . Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.6.1. Scheduler service parameters Scheduler service parameters are provided for the configuration of the cinderScheduler portions of the Block Storage service Parameter Description debug Provides a setting for the logging level. When this parameter is true the logging level is set to DEBUG instead of INFO . The default is false . scheduler_max_attempts Provides a setting for the maximum number of attempts to schedule a volume. The default is 3 scheduler_default_filters Provides a setting for filter class names to use for filtering hosts when not specified in the request. This is a comma separated list. The default is AvailabilityZoneFilter,CapacityFilter,CapabilitiesFilter . scheduler_default_weighers Provide a setting for weigher class names to use for weighing hosts. This is a comma separated list. The default is CapacityWeigher . scheduler_weight_handler Provides a setting for a handler to use for selecting the host or pool after weighing. The value cinder.scheduler.weights.OrderedHostWeightHandler selects the first host from the list of hosts that passed filtering and the value cinder.scheduler.weights.stochastic.stochasticHostWeightHandler gives every pool a chance to be chosen where the probability is proportional to each pool weight. The default is cinder.scheduler.weights.OrderedHostWeightHandler . The following is an explanation of the filter class names from the parameter table: AvailabilityZoneFilter Filters out all back ends that do not meet the availability zone requirements of the requested volume. CapacityFilter Selects only back ends with enough space to accommodate the volume. CapabilitiesFilter Selects only back ends that can support any specified settings in the volume. InstanceLocality Configures clusters to use volumes local to the same node. 4.7. Configuring the volume service The Block Storage service (cinder) has a volume service ( cinderVolumes section) that is responsible for managing operations related to volumes, snapshots, and groups. These operations include creating, deleting, and cloning volumes and making snapshots. This service requires access to the storage back end ( storage ) and storage management ( storageMgmt ) networks in the networkAttachments of the OpenStackControlPlane CR. Some operations, such as creating an empty volume or a snapshot, does not require any data movement between the volume service and the storage back end. Other operations though, such as migrating data from one storage back end to another, that requires the data to pass through the volume service to do require access. Volume service configuration is performed in the cinderVolumes section with parameters set in the customServiceConfig , customServiceConfigSecrets , networkAttachments , replicas , and the nodeSelector sections. The volume service cannot have multiple replicas. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the configuration for your back end. The following example demonstrates the service configuration for a Red Hat Ceph Storage back end: 1 The configuration area for the individual back end. Each unique back end requires an individual configuration area. No back end is deployed by default. The Block Storage service volume service will not run unless at least one back end is configured during deployment. For more information about configuring back ends, see Block Storage service (cinder) back ends and Multiple Block Storage service (cinder) back ends . 2 The configuration area for the back end network connections. 3 The name assigned to this back end. 4 The driver used to connect to this back end. For a list of commonly used volume service parameters, see Volume service parameters . Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.7.1. Volume service parameters Volume service parameters are provided for the configuration of the cinderVolumes portions of the Block Storage service Parameter Description backend_availability_zone Provides a setting for the availability zone of the back end. This is set in the [DEFAULT] section. The default value is storage_availability_zone . volume_backend_name Provides a setting for the back end name for a given driver implementation. There is no default value. volume_driver Provides a setting for the driver to use for volume creation. It is provided in the form of Python namespace for the specific class. There is no default value. enabled_backends Provides a setting for a list of back end names to use. These back end names should be backed by a unique [CONFIG] group with its options. This is a comma-seperated list of values. The default value is the name of the section with a volume_backend_name option. image_conversion_dir Provides a setting for a directory used for temporary storage during image conversion. The default value is /var/lib/cinder/conversion . backend_stats_polling_interval Provides a setting for the number of seconds between the volume requests for usage statistics from the storage back end. The default is 60 . 4.7.2. Block Storage service (cinder) back ends Each Block Storage service back end should have an individual configuration section in the cinderVolumes section. This ensures each back end runs in a dedicated pod. This approach has the following benefits: Increased isolation. Adding and removing back ends is fast and does not affect other running back ends. Configuration changes do not affect other running back ends. Automatically spreads the Volume pods into different nodes. Each Block Storage service back end uses a storage transport protocol to access data in the volumes. Each storage transport protocol has individual requirements as described in Configuring transport protocols . Storage protocol information should also be provided in individual vendor installation guides. Note Configure each back end with an independent pod. In director-based releases of RHOSP, all back ends run in a single cinder-volume container. This is no longer the best practice. No back end is deployed by default. The Block Storage service volume service will not run unless at least one back end is configured during deployment. All storage vendors provide an installation guide with best practices, deployment configuration, and configuration options for vendor drivers. These installation guides provide the specific configuration information required to properly configure the volume service for deployment. Installation guides are available in the Red Hat Ecosystem Catalog . For more information on integrating and certifying vendor drivers, see Integrating partner content . For information on Red Hat Ceph Storage back end configuration, see Integrating Red Hat Ceph Storage and Deploying a Hyperconverged Infrastructure environment . For information on configuring a generic (non-vendor specific) NFS back end, see Configuring a generic NFS back end . Note Use a certified storage back end and driver. If you use NFS storage that comes from the generic NFS back end, its capabilities are limited compared to a certified storage back end and driver. 4.7.3. Multiple Block Storage service (cinder) back ends Multiple Block Storage service back ends are deployed by adding multiple, independent entries in the cinderVolumes configuration section. Each back end runs in an independent pod. The following configuration example, deploys two independent back ends; one for iSCSI and another for NFS: 4.8. Configuring back end availability zones Configure back end availability zones (AZs) for Volume service back ends and the Backup service to group cloud infrastructure services for users. AZs are mapped to failure domains and Compute resources for high availability, fault tolerance, and resource scheduling. For example, you could create an AZ of Compute nodes with specific hardware that users can select when they create an instance that requires that hardware. Note Post-deployment, AZs are created using the RESKEY:availability_zones volume type extra specification. Users can create a volume directly in an AZ as long as the volume type does not restrict the AZ. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the AZ configuration. The following example demonstrates an AZ configuration: 1 The availability zone associated with the back end. Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.9. Configuring a generic NFS back end The Block Storage service (cinder) can be configured with a generic NFS back end to provide an alternative storage solution for volumes and backups. The Block Storage service supports a generic NFS solution with the following caveats: Use a certified storage back end and driver. If you use NFS storage that comes from the generic NFS back end, its capabilities are limited compared to a certified storage back end and driver. For example, the generic NFS back end does not support features such as volume encryption and volume multi-attach. For information about supported drivers, see the Red Hat Ecosystem Catalog . For Block Storage (cinder) and Compute (nova) services, you must use NFS version 4.0 or later. RHOSO does not support earlier versions of NFS. RHOSO does not support the NetApp NAS secure feature. It interferes with normal volume operations. This feature must be disabled in the customServiceConfig in the specific back-end configuration with the following parameters: Do not configure the nfs_mount_options option. The default value is the best NFS options for RHOSO environments. If you experience issues when you configure multiple services to share the same NFS server, contact Red Hat Support. Procedure Create a Secret CR to store the volume connection information. The following is an example of a Secret CR: 1 The name used when including it in the cinderVolumes back end configuration. Save the file. Update the control plane: Replace <secret_file_name> with the name of the file that contains your Secret CR. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the configuration for the generic NFS back end. The following example demonstrates this configuration: 1 The storageMgmt network is not listed because generic NFS does not have a management interface. 2 The name from the Secret CR. Note If you are configuring multiple generic NFS back ends, ensure each is in an individual configuration section so that one pod is devoted to each back end. Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.10. Configuring an NFS conversion directory When the Block Storage service (cinder) performs image format conversion, and the space is limited, conversion of large Image service (glance) images can cause the node root disk space to be completely used. You can use an external NFS share for the conversion to prevent the space on the node from being completely filled. Procedure Open your OpenStackControlPlane CR file, openstack_control_plane.yaml . Edit the CR file and add the configuration for the conversion directory. The following example demonstrates a conversion directory configuration: 1 The path to the directory to use for conversion. 2 The IP address of the server providing the conversion directory. Note The example provided demonstrates how to create a common conversion directory used by all volume service pods. It is also possible to define a conversion directory for each volume service pod. To do this, define each conversion directory using extraMounts as demonstrated above but in the cinder section of the OpenStackControlPlane CR file. You would then set the propagation value to the name of the specific Volume section instead of CinderVolume . Save the file. Update the control plane: Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: The OpenStackControlPlane resources are created when the status is "Setup complete". Tip Append the -w option to the end of the get command to track deployment progress. 4.11. Configuring automatic database cleanup The Block Storage service (cinder) performs a soft-deletion of database entries. This means that database entries are marked for deletion but are not actually deleted from the database. This allows for the auditing of deleted resources. These database rows marked for deletion will grow endlessly and consume resources if not purged. RHOSO automatically purges database entries marked for deletion after a set number of days. By default, records marked for deletion after 30 days are purged. You can configure a different record age and schedule for purge jobs. Procedure Open your openstack_control_plane.yaml file to edit the OpenStackControlPlane CR. Add the dbPurge parameter to the cinder template to configure database cleanup depending on the service you want to configure. The following is an example of using the dbPurge parameter to configure the Block Storage service: 1 The number of days a record has been marked for deletion before it is purged. The default value is 30 . The minimum value is 1 . 2 The schedule of when to run the job in a crontab format. The default value is 1 0 * * * . This default value is equivalent to 00:01 daily. Update the control plane: 4.12. Preserving jobs The Block Storage service (cinder) requires maintenance operations that are run automatically. Some operations are one-off and some are periodic. These operations are run using OpenShift Jobs. If jobs and their pods are automatically removed on completion, you cannot check the logs of these operations. However, you can use the preserveJob field in your OpenStackControlPlane CR to stop the automatic removal of jobs and preserve them. Example: 4.13. Resolving hostname conflicts Most storage back ends in the Block Storage service (cinder) require the hosts that connect to them to have unique hostnames. These hostnames are used to identify permissions and addresses, such as iSCSI initiator name, HBA WWN and WWPN. Because you deploy in OpenShift, the hostnames that the Block Storage service volumes and backups report are not the OpenShift hostnames but the pod names instead. These pod names are formed using a predetermined template: * For volumes: cinder-volume-<backend_key>-0 * For backups: cinder-backup-<replica-number> If you use the same storage back end in multiple deployments, the unique hostname requirement may not be honored, resulting in operational problems. To address this issue, you can request the installer to have unique pod names, and hence unique hostnames, by using the uniquePodNames field. When you set the uniquePodNames field to true , a short hash is added to the pod names, which addresses hostname conflicts. Example: 4.14. Using other container images Red Hat OpenStack Services on OpenShift (RHOSO) services are deployed using a container image for a specific release and version. There are times when a deployment requires a container image other than the one produced for that release and version. The most common reasons for this are: Deploying a hotfix. Using a certified, vendor-provided container image. The container images used by the installer are controlled through the OpenStackVersion CR. An OpenStackVersion CR is automatically created by the openstack operator during the deployment of services. Alternatively, it can be created manually before the application of the OpenStackControlPlane CR but after the openstack operator is installed. This allows for the container image for any service and component to be individually designated. The granularity of this designation depends on the service. For example, in the Block Storage service (cinder) all the cinderAPI , cinderScheduler , and cinderBackup pods must have the same image. However, for the Volume service, the container image is defined for each of the cinderVolumes . The following example demonstrates a OpenStackControlPlane configuration with two back ends; one called ceph and one called custom-fc . The custom-fc backend requires a certified, vendor-provided container image. Additionally, we must configure the other service images to use a non-standard image from a hotfix. The following example demonstrates what our OpenStackVersion CR might look like in order to set up the container images properly. Replace <custom-api-image> with the name of the API service image to use. Replace <custom-backup-image> with the name of the Backup service image to use. Replace <custom-scheduler-image> with the name of the Scheduler service image to use. Replace <vendor-volume-volume-image> with the name of the certified, vendor-provided image to use. Note The name attribute in your OpenStackVersion CR must match the same attribute in your OpenStackControlPlane CR.

Commands:
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder:
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: <global-options> template: <global-options> cinderAPI: <cinder-api-options> cinderScheduler: <cinder-scheduler-options> cinderVolumes: <name1>: <cinder-volume-options> <name2>: <cinder-volume-options> cinderBackup: <cinder-backup-options>
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-worker-cinder-enable-iscsid spec: config: ignition: version: 3.2.0 systemd: units: - enabled: true name: iscsid.service
oc apply -f <machine_config_file> -n openstack
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-worker-cinder-load-nvme-fabrics spec: config: ignition: version: 3.2.0 storage: files: - path: /etc/modules-load.d/nvme_fabrics.conf overwrite: false mode: 420 user: name: root group: name: root contents: source: data:,nvme-fabrics%0Anvme-tcp
oc apply -f <machine_config_file> -n openstack
cat /sys/module/nvme_core/parameters/multipath
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-worker-cinder-enable-multipathd spec: config: ignition: version: 3.2.0 storage: files: - path: /etc/multipath.conf overwrite: false mode: 384 user: name: root group: name: root contents: source: data:,defaults%20%7B%0A%20%20user_friendly_names%20no%0A%20%20recheck_wwid%20yes%0A%20%20skip_kpartx%20yes%0A%20%20find_multipaths%20yes%0A%7D%0A%0Ablacklist%20%7B%0A%7D systemd: units: - enabled: true name: multipathd.service
defaults { user_friendly_names no recheck_wwid yes skip_kpartx yes find_multipaths yes } blacklist { }
oc apply -f <machine_config_file> -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: enabled: true template: customServiceConfig: | [DEFAULT] quota_volumes = 20 quota_snapshots = 15
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderAPI: override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.80 spec: type: LoadBalancer
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderAPI: replicas: 3
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: customServiceConfig: | [DEFAULT] debug = true cinderAPI: customServiceConfig: | [DEFAULT] osapi_volume_workers = 3
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: customServiceConfig: | [DEFAULT] report_interval = 20 1 service_down_time = 120 2
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: customServiceConfig: | [DEFAULT] backend_stats_polling_interval = 120 1 backup_driver_stats_polling_interval = 120 2
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderBackup: customServiceConfig: | [DEFAULT] backup_driver_stats_polling_interval = 120 1 < rest of the config > cinderVolumes: nfs: customServiceConfig: | [DEFAULT] backend_stats_polling_interval = 120 2
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: customServiceConfig: | [DEFAULT] debug = true cinderVolumes: ceph: 1 networkAttachments: 2 - storage customServiceConfig: | [ceph] volume_backend_name = ceph 3 volume_driver = cinder.volume.drivers.rbd.RBDDriver 4
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderVolumes: nfs: networkAttachments: - storage customServiceConfigSecrets: - cinder-volume-nfs-secrets customServiceConfig: | [nfs] volume_backend_name=nfs iSCSI: networkAttachments: - storage - storageMgmt customServiceConfig: | [iscsi] volume_backend_name=iscsi
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderVolumes: nfs: networkAttachments: - storage - storageMgmt customServiceConfigSecrets: - cinder-volume-nfs-secrets customServiceConfig: | [nfs] volume_backend_name=nfs backend_availability_zone=zone1 1 iSCSI: networkAttachments: - storage - storageMgmt customServiceConfig: | [iscsi] volume_backend_name=iscsi backend_availability_zone=zone2
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
nas_secure_file_operation=false nas_secure_file_permissions=false
apiVersion: v1 kind: Secret metadata: name: cinder-volume-nfs-secrets 1 type: Opaque stringData: cinder-volume-nfs-secrets: | [nfs] nas_host=192.168.130.1 nas_share_path=/var/nfs/cinder
oc apply -f <secret_file_name> -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderVolumes: nfs: networkAttachments: 1 - storage customServiceConfig: | [nfs] volume_backend_name=nfs volume_driver=cinder.volume.drivers.nfs.NfsDriver nfs_snapshot_support=true nas_secure_file_operations=false nas_secure_file_permissions=false customServiceConfigSecrets: - cinder-volume-nfs-secrets 2
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane spec: extraMounts: extraVol: - propagation: - CinderVolume volumes: - name: cinder-conversion nfs: path: <nfs_share_path> 1 server: <nfs_server> 2 mounts: - name: cinder-conversion mountPath: /var/lib/cinder/conversion readOnly: true
oc apply -f openstack_control_plane.yaml -n openstack
oc get openstackcontrolplane -n openstack
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: dbPurge: age: 20 1 schedule: 1 0 * * 0 2
oc apply -f openstack_control_plane.yaml
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: preserveJobs: true
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: uniquePodNames: true
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderVolumes: ceph: networkAttachments: - storage < . . . > custom-fc: networkAttachments: - storage
apiVersion: core.openstack.org/v1beta1 kind: OpenStackVersion metadata: name: openstack spec: customContainerImages: cinderAPIImages: <custom-api-image> cinderBackupImages: <custom-backup-image> cinderSchedulerImages: <custom-scheduler-image> cinderVolumeImages: custom-fc: <vendor-volume-volume-image>