Chapter 7. Planning storage and shared file systems Red Hat OpenStack Services on OpenShift (RHOSO) uses ephemeral and persistent storage to service the storage needs of the deployment. Ephemeral storage is associated with a specific Compute instance. When this instance is terminated, so is the associated ephemeral storage. Ephemeral storage is useful for runtime requirements, such as storing the operating system of an instance. Persistent storage is independent of any running instance. Persistent storage is useful for storing reusable data, such as data volumes, disk images, and shareable file systems. The storage requirements of the deployment should be taken into consideration and carefully planned before beginning the deployment. This includes considerations such as: Supported features and topologies Storage technologies Networking Scalability Accessibility Performances Costs Security Redundancy and disaster recovery Storage management 7.1. Supported storage features and topologies RHOSO supports the following storage and networking features: Red Hat Ceph Storage integration: Ceph Block Device (RBD) with the Block Storage service (cinder) for persistent storage, the Image service (glance), and the Compute service (nova) for ephemeral storage. Ceph File System (Native CephFS or CephFS via NFS) with the Shared File Systems service (manila). Object Storage service integration with Ceph Object Gateway (RGW) Hyperconverged infrastructure (HCI): Hyperconverged infrastructures consist of hyperconverged nodes. Hyperconverged nodes are external data plane nodes with Compute and Red Hat Ceph Storage services colocated on the same nodes for optimized hardware footprint. Transport protocols for the Block Storage service with appropriate configuration and drivers: NVMe over TCP RBD NFS FC Note You must install host bus adapters (HBAs) on all Compute and OCP workers nodes in any deployment that uses the Block Storage service and a Fibre Channel (FC) back end. iSCSI Multipathing with iSCSI, FC, and NVMe over TCP is available on the control plane with the appropriate RHOCP MachineConfig. Transport protocols for the Shared File Systems service with appropriate configuration and drivers: CephFS NFS CIFS Object Storage through native Swift or Amazon S3 compatible API RHOSO supports the following storage services. Service Back ends Image service (glance) Red Hat Ceph Storage RBD Block Storage (cinder) Object Storage (swift) NFS Compute service (nova) local file storage Red Hat Ceph Storage RBD Block Storage service (cinder) Red Hat Ceph Storage RBD Fiber Channel iSCSI NFS NVMe over TCP Note Support is provided through third party drivers. Shared File Systems service (manila) Red Hat Ceph Storage CephFS Red Hat Ceph Storage CephFS-NFS NFS or CIFS through third party vendor storage systems Object Storage service (swift) disks on external data plane nodes PersistentVolumes (PVs) on OpenShift nodes (default) Integration with Ceph RGW To manage the consumption of system resources by projects, you can configure quotas for the Block Storage service (cinder) and the Shared File Systems service (manila). You can override the default quotas so that individual projects have different consumption limits. 7.2. Storage technologies RHOSO supports a number of storage technologies that can act separately or in combination to provide the storage solution for your deployment. 7.2.1. Red Hat Ceph Storage Red Hat Ceph Storage is a distributed data object store designed for performance, reliability, and scalability. Distributed object stores use unstructured data to simultaneously service modern and legacy object interfaces. It provides access to block, file, and object storage. Red Hat Ceph Storage is deployed as a cluster. A cluster consists of two primary types of daemons: Ceph Object Storage Daemon (CephOSD) - The CephOSD performs data storage, data replication, rebalancing, recovery, monitoring, and reporting tasks. Ceph Monitor (CephMon) - The CephMon maintains the primary copy of the cluster map with the current state of the cluster. RHOSO supports Red Hat Ceph Storage 7 in the following deployment scenarios: Integration with an externally deployed Red Hat Ceph Storage 7 cluster. A hyperconverged infrastructure (HCI) environment that consists of external data plane nodes that have Compute and Red Hat Ceph Storage services colocated on the same nodes for optimized resource use. Note Red Hat OpenStack Services on OpenShift (RHOSO) 18.0 supports erasure coding with Red Hat Ceph Storage Object Gateway (RGW). Erasure coding with the Red Hat Ceph Storage Block Device (RDB) is not currently supported. For more information about Red Hat Ceph Storage architecture, see the Red Hat Ceph Storage 7 Architecture Guide . 7.2.2. Block storage (cinder) The Block Storage service (cinder) allows users to provision block storage volumes on back ends. Users can attach volumes to instances to augment their ephemeral storage with general-purpose persistent storage. You can detach and re-attach volumes to instances, but you can only access these volumes through the attached instance. You can also configure instances so that they do not use ephemeral storage. Instead of using ephemeral storage, you can configure the Block Storage service to write images to a volume. You can then use the volume as a bootable root volume for an instance. Volumes also provide inherent redundancy and disaster recovery through backups and snapshots. However, backups are only provided if you deploy the optional Block Storage backup service. In addition, you can encrypt volumes for added security. 7.2.3. Images (glance) The Image service (glance) provides discovery, registration, and delivery services for instance images. It also provides the ability to store snapshots of instances ephemeral disks for cloning or restore purposes. You can use stored images as templates to commission new servers quickly and more consistently than installing a server operating system and individually configuring services. 7.2.4. Object Storage (swift) The Object Storage service (swift) provides a fully-distributed storage solution that you can use to store any kind of static data or binary object; such as media files, large datasets, and disk images. The Object Storage service organizes objects by using object containers, which are similar to directories in a file system, but they cannot be nested. You can use the Object Storage service as a repository for nearly every service in the cloud. Red Hat Ceph Storage RGW can be used as an alternative to the Object Storage service. 7.2.5. Shared File Systems (manila) The Shared File Systems service (manila) provides the means to provision remote, shareable file systems. These are known as shares. Shares allow projects in the cloud to share POSIX compliant storage, and they can be consumed by multiple instances simultaneously. Shares are used for instance consumption, and they can be consumed by multiple instances at the same time with read/write access mode. 7.3. Storage networks Two default storage-related networks are configured during the RHOSO installation: the Storage and Storage Management networks. These isolated networks follow best practices for network connectivity between storage components and the deployments. The Storage network is used for data storage access and retrieval. The Storage Management network is used by RHOSO services to have access to specific interfaces in the storage solution that allows access to the management consoles. For example, Red Hat Ceph Storage uses the Storage Management network in a hyperconverged infrastructure (HCI) environment as the cluster_network to replicate data. The following table lists the properties of the default storage-related networks. Network name VLAN CIDR NetConfig allocationRange MetalLB IPAddressPool range nad ipam range OCP worker nncp range storage 21 172.18.0.0/24 172.18.0.100 - 172.18.0.250 N/A 172.18.0.30 - 172.18.0.70 172.18.0.10 - 172.18.0.20 storageMgmt 23 172.20.0.0/24 172.20.0.100 - 172.20.0.250 N/A 172.20.0.30 - 172.20.0.70 172.20.0.10 - 172.20.0.20 Your storage solution may require additional network configurations. These defaults provide a basis for building a full deployment. All Block Storage services with back ends ( cinder-volume and cinder-backup ) require access to all the storage networks, which may not include the storage management network depending on the back end. Block Storage services with back ends require access only to their storage management network. In most deployments there's a single management network, but if there are multiple storage management networks, each service-back end pair only needs access to their respective management network. You must install host bus adapters (HBAs) on all OCP worker nodes in any deployment that uses the Block Storage service and a Fibre Channel (FC) back end. 7.3.1. Planning networking for the Block Storage service Storage best practices recommend using two different networks: One network for data I/O One network for storage management These networks are referred to as storage and storageMgmt . If your deployment diverges from the architecture of two networks, adapt the documented examples as necessary. For example, if the management interface for the storage system is available on the storage network, replace storageMgmt with storage when there is only one network, and remove storageMgmt when the storage network is already present. The storage services in Red Hat OpenStack Services on OpenShift (RHOSO), with the exception of the Object Storage service (swift), require access to the storage and storageMgmt networks. You can configure the storage and storageMgmt networks in the networkAttachments field of your OpenStackControlPlane CR. The networkAttachments field accepts a list of strings with all the networks the component requires access to. Different components can have different network requirements, for example, the Block Storage service (cinder) API component does not require access to any of the storage networks. The following example shows the networkAttachments for Block Storage volumes: 7.3.2. Planning networking for the Shared File Systems service Plan the networking on your cloud to ensure that cloud users can connect their shares to workloads that run on Red Hat OpenStack Services on OpenShift (RHOSO) virtual machines, bare-metal servers, and containers. Depending on the level of security and isolation required for cloud users, you can set the driver_handles_share_servers parameter (DHSS) to true or false . 7.3.2.1. Setting DHSS to true If you set the DHSS parameter to true , you can use the Shared File Systems service to export shares to end-user defined share networks with isolated share servers. Users can provision their workloads on self-service share networks to ensure that isolated NAS file servers on dedicated network segments export their shares. As a project administrator, you must ensure that the physical network to which you map the isolated networks extends to your storage infrastructure. You must also ensure that the storage system that you are using supports network segments. Storage systems, such as NetApp ONTAP and Dell EMC PowerMax, Unity, and VNX, do not support virtual overlay segmentation styles such as GENEVE or VXLAN. As an alternative to overlay networking, you can do any of the following: Use VLAN networking for your project networks. Allow VLAN segments on shared provider networks. Provide access to a pre-existing segmented network that is already connected to your storage system. 7.3.2.2. Setting DHSS to false If you set the DHSS parameter to false , cloud users cannot create shares on their own share networks. You can create a dedicated shared storage network, and cloud users must connect their clients to the configured network to access their shares. Not all Shared File System storage drivers support both DHSS=true and DHSS=false . Both DHSS=true and DHSS=false ensure data path multi-tenancy isolation. However, if you require network path multi-tenancy isolation for tenant workloads as part of a self-service model, you must deploy the Shared File Systems service (manila) with back ends that support DHSS=true . 7.3.2.3. Ensuring network connectivity to the share To connect to a file share, clients must have network connectivity to one or more of the export locations for that share. When administrators set the driver_handles_share_servers parameter (DHSS) for a share type to true , cloud users can create a share network with the details of a network to which the Compute instance attaches. Cloud users can then reference the share network when creating shares. When administrators set the DHSS parameter for a share type to false , cloud users must connect their Compute instance to the shared storage network that has been configured for their Red Hat OpenStack Services on OpenShift (RHOSO) deployment. For more information about how to configure and validate network connectivity to a shared network, see Connecting to a shared network to access shares in Performing storage operations . 7.4. Scalability and back-end storage In general, a clustered storage solution provides greater back end scalability and resiliency. For example, when you use Red Hat Ceph Storage as a Block Storage (cinder) back end, you can scale storage capacity and redundancy by adding more Ceph Object Storage Daemon (OSD) nodes. Block Storage, Object Storage (swift), and Shared File Systems Storage (manila) services support Red Hat Ceph Storage as a back end. The Block Storage service can use multiple storage solutions as discrete back ends. At the service level, you can scale capacity by adding more back ends. By default, the Object Storage service consumes space by allocating persistent volumes in the OpenShift underlying infrastructure. It can be configured to use a file system on dedicated storage nodes, and it can use as much space as is available. The Object Storage service supports the XFS and ext4 file systems, and you can scale both file systems to consume as much underlying block storage as is available. You can also scale capacity by adding more storage devices to the storage node. The Shared File Systems service provisions file shares from designated storage pools that are managed by Red Hat Ceph Storage or other back-end storage systems. You can scale this shared storage by increasing the size or number of storage pools available to the service or by adding more back-end storage systems to the deployment. Each back-end storage system is integrated with a dedicated service to interact with and manage the storage system. 7.5. Storage accessibility and administration Volumes are consumed only through instances. Users can extend, create snapshots of volumes and use the snapshots to clone or restore a volume to a state. You can use the Block Storage service (cinder) to create volume types, which aggregate volume settings. You can associate volume types with encryption and Quality of Service (QoS) specifications to provide different levels of performance for your cloud users. Your cloud users can specify the volume type they require when creating new volumes. For example, volumes that use higher performance QoS specifications could provide your users with more IOPS, or your users could assign lighter workloads to volumes that use lower performance QoS specifications to conserve resources. Shares can be consumed simultaneously by one or more instances, bare metal nodes or containers. The Shared File Systems service (manila) also supports share resize, snapshots and cloning, and administrators can create share types to aggregate settings. Users can access objects in a container by using the Object Storage service (swift) API, and administrators can make objects accessible to instances and services in the cloud. This accessibility makes objects ideal as repositories for services; for example, you can store Image service (glance) images in containers that are managed by the Object Storage service. 7.6. Storage security The Block Storage service provides data security through the Key Manager service (barbican). The Block Storage service uses a one-to-one, key to volume mapping with the key managed by the Key Manager service. The encryption type is defined when configuring the volume type. Security can also be improved at the backend level by encrypting control and/or data traffic, for example with Red Hat Ceph Storage, this can be achieved by enabling messengerv2 secure mode. This way, network traffic amongst Ceph services as well as from OpenStack compute nodes are encrypted. You configure object and container security at the service and node level. The Object Storage service (swift) provides no native encryption for containers and objects. However, with the Key Manager service enabled, the Object Storage service can transparently encrypt and decrypt your stored (at-rest) objects. At-rest encryption is distinct from in-transit encryption in that it refers to the objects being encrypted while being stored on disk. The Shared File Systems service (manila) can secure shares through access restriction, whether by instance IP, user or group, or TLS certificate. Some Shared File Systems service deployments can feature separate share servers to manage the relationship between share networks and shares. Some share servers support, or even require, additional network security. For example, a CIFS share server requires the deployment of an LDAP, Active Directory, or Kerberos authentication service. Some backends also support encrypting the data AT REST. This enables extra security by encrypting the backend disks themselves, preventing physical security threats such as theft or unwiped recycled disks. For more information about configuring security options for the Block Storage service, Object Storage service, and Shared File Systems service, see Configuring security services . 7.7. Storage redundancy and disaster recovery If you deploy the optional Block Storage backup service, then the Block Storage service (cinder) provides volume backup and restoration for basic disaster recovery of user storage. You can use backups to protect volume contents. The Block Storage service also supports snapshots. In addition to cloning, you can use snapshots to restore a volume to a state. If your environment includes multiple back ends, you can also migrate volumes between these back ends. This is useful if you need to take a back end offline for maintenance. Backups are typically stored in a storage back end separate from their source volumes to help protect the data. This is not possible with snapshots because snapshots are dependent on their source volumes. The Block Storage service also supports the creation of consistency groups to group volumes together for simultaneous snapshot creation. This provides a greater level of data consistency across multiple volumes. Note Red Hat does not currently support Block Storage service replication. The Object Storage service (swift) provides no built-in backup features. You must perform all backups at the file system or node level. However, the Object Storage service features robust redundancy and fault tolerance. Even the most basic deployment of the Object Storage service replicates objects multiple times. You can use failover features like device mapper multipathing (DM Multipath) to enhance redundancy. The Shared File Systems service (manila) provides no built-in backup features for shares, but you can create snapshots for cloning and restoration. 7.8. Managing the storage solution You can manage your RHOSO configuration using the RHOSO Dashboard (horizon) or the RHOSO command line interface (CLI). You can perform most procedures using either method but some advanced procedures can only be completed using the CLI. You can manage your storage solution configuration using the dedicated management interface provided by the storage vendor. 7.9. Sizing Red Hat OpenShift storage The Image and Object Storage services can be configured to allocate space in the Red Hat OpenShift backing storage. In this scenario, the Red Hat OpenShift storage sizing should be estimated based on the expected use of these services. 7.9.1. Image service considerations The Image service (glance) requires a staging area to manipulate data during an import operation. It is possible to copy image data into multiple stores so some persistence is required for the Image service. Although PVCs represent the main storage model for the Image service, an External model can also be chosen. External model If External is chosen, no PVCs are created and the Image service acts like a stateless instance with no persistence provided. In this instance, persistence must be provided using extraMounts . NFS is often used to provide persistence. It can be mapped to /var/lib/glance : Replace <nfs_export_path> with the export path of your NFS share. Replace <nfs_ip_address> with the IP address of your NFS share. This IP address must be part of the overlay network that is reachable by the Image service. It should be noted that the configuration sample conflicts with the distributed image import feature. Distributed image import requires RWO storage plugged into a particular instance; it owns the data and receives requests in case its staged data is required for an upload operation. When the External model is adopted, if Red Hat Ceph Storage is used as a backend, and an image conversion operation is run in one of the existing replicas, the glance-operator does not have to make any assumption about the underlying storage that is tied to the staging area, and the conversion operation that uses the os_glance_staging_store directory (within the Pod) interacts with the RWX NFS backend provided via extraMounts . With this scenario, no image-cache PVC can be requested and mounted to a subPath, because it should be the administrator's responsibility to plan for persistence using extraMounts . PVC model The PVC model is the default. When a GlanceAPI instance is deployed, a PVC is created and bound to /var/lib/glance according to the storageClass and storageRequest passed as input. In this model, if Red Hat Ceph Storage is set as a backend, no dedicated image conversion PVC is created. The administrator must think about the PVC sizing in advance; the size of the PVC should be at least up to the largest converted image size. Concurrent conversions within the same Pod might be problematic in terms of PVC size; a conversion will fail or cannot take place if the PVC is full and there's not enough space. The upload should be retried after the conversion is over and the staging area space is released. However, concurrent conversion operations might happen in different Pods. You should deploy at least 3 replicas for a particular glanceAPI . This helps to handle heavy operations like image conversion. For a PVC-based layout, the scale out of a glanceAPI in terms of replicas is limited by the available storage provided by the storageClass , and depends on the storageRequest . The storageRequest is a critical parameter, it can be globally defined for all the glanceAPI , or defined with a different value for each API. It will influence the scale out operations for each of them. Other than a local PVC required for the staging area, it is possible to enable image cache, which is translated into an additional PVC bound to each glanceAPI instance. A glance-cache PVC is bound to /var/lib/glance/image-cache . The glance-operator configures the glanceAPI instance accordingly, setting both image_cache_max_size and the image_cache_dir parameters. The number of image cache PVCs follows the same rules described for the local PVC, and the number of requested PVCs is proportional to the number of replicas. 7.9.2. Object Storage service considerations The Object Storage service requires storage devices for data. These devices must be accessible using the same hostname or IP address during their lifetime. The configuration of a StatefulSet with a Headless Service is how this is achieved. If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed. The Object Storage service requires quite a few services to access these PVs, and all of them are running in a single pod. Additionally, volumes are not deleted if the StatefulSet is deleted. An unwanted removal of the StatefulSet (or the whole deployment) will not immediately result in a catastrophic data loss, but can be recovered from with administrator interaction. The Headless Service makes it possible to access the storage pod directly by using a DNS name. For example, if the pod name is swift-storage-0 and the SwiftStorage instance is named swift-storage , it becomes accessible using swift-storage-0.swift-storage . This makes it easily usable within the Object Storage service rings, and IP changes are now transparent and don't require an update of the rings. Parallel pod management tells the StatefulSet controller to launch or terminate all Pods in parallel, and to not wait for Pods to become Running and Ready or completely terminated prior to launching or terminating another Pod. This option only affects the behavior for scaling operations. Updates are not affected. This is required to scale by more than one; including new deployments with more than one replica. It is required to create all pods at the same time, otherwise there will be PVCs that are not bound and the Object Storage service rings cannot be created, eventually blocking the start of these pods. Storage pods should be distributed to different nodes to avoid single points of failure. A podAntiAffinity rule with preferredDuringSchedulingIgnoredDuringExecution is used to distribute pods to different nodes if possible. Using a separate storageClass and PersistentVolumes that are located on different nodes can be used to enforce further distribution. Object Storage service backend services must only be accessible by other backend services and the Object Storage service proxy. To limit access, a NetworkPolicy is added to allow only traffic between these pods. The NetworkPolicy itself depends on labels, and these must match to allow traffic. Therefore labels must not be unique; instead all pods must use the same label to allow access. This is also the reason why the swift-operator is not using labels from lib-common . Object Storage service rings require information about the disks to use, and this includes sizes and hostnames or IPs. Sizes are not known when starting the StatefulSet using PVCs, the size requirement is a lower limit, but the actual PVs might be much bigger. However, StatefulSets do create PVCs before the ConfigMaps are available and simply wait starting the pods until these become available. The SwiftRing reconciler is watching the SwiftStorage instances and iterates over PVCs to get actual information about the used disks. Once these are bound, the size is known and the swift-ring-rebalance job creates the Swift rings and eventually the ConfigMap . After the ConfigMap becomes available, StatefulSets will start the service pods. Rings are stored in a ConfigMap mounted by the SwiftProxy and SwiftStorage instances using projected volumes. This makes it possible to mount all required files at the same place, without merging these from other places. Updated ConfigMaps will update these files, and these changes are are detected by the Swift services eventually reloading these. Some operators are using the customServiceConfig option to customize settings. However, the SwiftRing instance deploys multiple backend services, and each of these requires specific files to be customized. Therefore only defaultConfigOverwrite using specific keys as filenames is supported when using the swift-operator .

Commands:
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: template: cinderVolumes: iscsi: networkAttachments: - storage - storageMgmt
default: storage: external: true extraMounts: - extraVol: - extraVolType: NFS mounts: - mountPath: /var/lib/glance/os_glance_staging_store name: nfs volumes: - name: nfs nfs: path: <nfs_export_path> server: <nfs_ip_address>
default: replicas: 3 storage: storageRequest: 10G