Chapter 2. CephFS through NFS installation 2.1. CephFS with NFS-Ganesha deployment A typical Ceph file system (CephFS) through NFS installation in a Red Hat OpenStack Platform (RHOSP) environment includes the following configurations: OpenStack Controller nodes running containerized Ceph metadata server (MDS), Ceph monitor (MON), manila, and NFS-Ganesha services. Some of these services can coexist on the same node or can have one or more dedicated nodes. Ceph storage cluster with containerized object storage daemons (OSDs) running on Ceph storage nodes. An isolated StorageNFS network that provides access from projects to the NFS-Ganesha services for NFS share provisioning. The Shared File Systems service (manila) provides APIs that allow the projects to request file system shares, which are fulfilled by driver modules. The driver for Red Hat CephFS, manila.share.drivers.cephfs.driver.CephFSDriver , means that you can use the Shared File Systems service as a CephFS as a back end. RHOSP director configures the driver to deploy the NFS-Ganesha gateway so that the CephFS shares are presented through the NFS 4.1 protocol. Using RHOSP director to deploy the Shared File Systems service with a CephFS back end on the overcloud automatically creates the required storage network defined in the heat template. For more information about network planning, see Overcloud networks in the Director Installation and Usage guide. Although you can manually configure the Shared File Systems service by editing its node /etc/manila/manila.conf file, RHOSP director can override any settings in future overcloud updates. The recommended method for configuring a Shared File System back end is through director. Note Currently, you can define only one CephFS back end at a time in director. CephFS through NFS 2.1.1. Requirements for CephFS through NFS CephFS through NFS requires a Red Hat OpenStack Platform (RHOSP) version 13 or later environment, which can be an existing or a new environment. For RHOSP versions 13, 14, and 15, CephFS works with Red Hat Ceph Storage (RHCS) version 3. For RHOSP version 16 or later, CephFS works with Red Hat Ceph Storage (RHCS) version 4.1 or later. For more information, see the Deploying an Overcloud with Containerized Red Hat Ceph Guide . Prerequisites You install the Shared File Systems service on Controller nodes, as is the default behavior. You install the NFS-Ganesha gateway service on Pacemaker cluster of the Controller node. You configure only a single instance of a CephFS back end to use the Shared File Systems service. You can use other non-CephFS back ends with the single CephFS back end. You use RHOSP director to create an extra network (StorageNFS) for the storage traffic. You configure a new RHCS version 4.1 or later cluster at the same time as CephFS through NFS. 2.1.2. File shares File shares are handled differently in the Shared File Systems service (manila), Ceph File System (CephFS), and Ceph through NFS. The Shared File Systems service provides shares. A share is an individual file system namespace and a unit of storage or sharing and a defined size for example, subdirectories with quotas. Shared file system storage enables multiple clients because the file system is configured before access is requested unlike block storage, which is configured when it is requested. With CephFS, a share is considered a directory with a defined quota and a layout that points to a particular storage pool or namespace. CephFS quotas limit the size of a directory to the size share that the Shared File Systems service creates. Access to Ceph shares is determined by MDS authentication capabilities. With CephFS through NFS, file shares are provisioned and accessed through the NFS protocol. The NFS protocol also handles security. 2.1.3. Isolated network used by CephFS through NFS CephFS through NFS deployments use an extra isolated network, StorageNFS. This network is deployed so users can mount shares over NFS on that network without accessing the Storage or Storage Management networks which are reserved for infrastructure traffic. For more information about isolating networks, see Basic network isolation in the Advanced Overcloud Customization guide. 2.2. Installing Red Hat OpenStack Platform with CephFS through NFS and a custom network_data file To install CephFS through NFS, complete the following procedures: Install the ceph-ansible package. See Section 2.2.1, "Installing the ceph-ansible package" Prepare the overcloud container images with the openstack overcloud image prepare command. See Section 2.2.2, "Preparing overcloud container images" Generate the custom roles file, roles_data.yaml , and network_data.yaml file. See Section 2.2.2.1, "Generating the custom roles file" Deploy Ceph, Shared File Systems service (manila), and CephFS using the openstack overcloud deploy command with custom roles and environments. See Section 2.2.3, "Deploying the updated environment" Configure the isolated StorageNFS network and create the default share type. See Section 2.2.4, "Completing post-deployment configuration" Examples use the standard stack user in the Red Hat Platform (RHOSP) environment. Perform tasks as part of a RHOSP installation or environment update. 2.2.1. Installing the ceph-ansible package Install the ceph-ansible package to be installed on an undercloud node to deploy containerized Ceph. Procedure Log in to an undercloud node as the stack user. Install the ceph-ansible package: 2.2.2. Preparing overcloud container images Because all services are containerized in Red Hat OpenStack Platform (RHOSP), you must prepare container images for the overcloud by using the openstack overcloud image prepare command. Enter this command with the additional options to add default images for the ceph and manila services to the container registry. Ceph MDS and NFS-Ganesha services use the same Ceph base container image. For more information about container images, see Container Images for Additional Services in the Director Installation and Usage guide. Procedure From the undercloud as the stack user, enter the openstack overcloud image prepare command with -e to include the following environment files: Use grep to verify that the default images for the ceph and manila services are available in the containers-default-parameters.yaml file. 2.2.2.1. Generating the custom roles file The ControllerStorageNFS custom role configures the isolated StorageNFS network. This role is similar to the default Controller.yaml role file with the addition of the StorageNFS network and the CephNfs service, indicated by the OS::TripleO::Services:CephNfs command. For more information about the openstack overcloud roles generate command, see Roles in the Advanced Overcloud Customization guide. The openstack overcloud roles generate command creates a custom roles_data.yaml file including the services specified after -o . In the following example, the roles_data.yaml file created has the services for ControllerStorageNfs , Compute , and CephStorage . Note If you have an existing roles_data.yaml file, modify it to add ControllerStorageNfs , Compute , and CephStorage services to the configuration file. For more information, see Roles in the Advanced Overcloud Customization guide. Procedure Log in to an undercloud node as the stack user, Use the openstack overcloud roles generate command to create the roles_data.yaml file: 2.2.3. Deploying the updated environment When you are ready to deploy your environment, use the openstack overcloud deploy command with the custom environments and roles required to run CephFS with NFS-Ganesha. The overcloud deploy command has the following options in addition to other required options. Action Option Additional information Add the updated default containers from the overcloud container image prepare command. -e /home/stack/containers-default-parameters.yaml` Section 2.2.2, "Preparing overcloud container images" Add the extra StorageNFS network with network_data_ganesha.yaml . -n /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml` Section 2.2.3.1, "StorageNFS and network_data_ganesha.yaml file" Add the custom roles defined in roles_data.yaml file from the section. -r /home/stack/roles_data.yaml . Section 2.2.2.1, "Generating the custom roles file" Deploy the Ceph daemons with ceph-ansible.yaml . -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml Initiating Overcloud Deployment in the Deploying an Overcloud with Containzerized Red Hat Ceph guide Deploy the Ceph metadata server with ceph-mds.yaml . -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-mds.yaml . Initiating Overcloud Deployment in the Deploying an Overcloud with Containzerized Red Hat Ceph guide Deploy the manila service with the CephFS through NFS back end. Configure NFS-Ganesha with director. -e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml Section 2.2.3.2, " manila-cephfsganesha-config.yaml " The following example shows an openstack overcloud deploy command with options to deploy CephFS through NFS-Ganesha, Ceph cluster, Ceph MDS, and the isolated StorageNFS network: For more information about the openstack overcloud deploy command, see Deployment command in the Director Installation and Usage guide. 2.2.3.1. StorageNFS and network_data_ganesha.yaml file Use composable networks to define custom networks and assign them to any role. Instead of using the standard network_data.yaml file, you can configure the StorageNFS composable network with the network_data_ganesha.yaml file. Both of these roles are available in the /usr/share/openstack-tripleo-heat-templates directory. The network_data_ganesha.yaml file contains an additional section that defines the isolated StorageNFS network. Although the default settings work for most installations, you must edit the YAML file to add your network settings, including the VLAN ID, subnet, and other settings. For more information about composable networks, see Using Composable Networks in the Advanced Overcloud Customization guide. 2.2.3.2. manila-cephfsganesha-config.yaml The integrated environment file for defining a CephFS back end is located in the following path of an undercloud node: The manila-cephfsganesha-config.yaml environment file contains settings relevant to the deployment of the Shared File Systems service. The back end default settings work for most environments. The following example shows the default values that director uses during deployment of the Shared File Systems service: The parameter_defaults header signifies the start of the configuration. In this section, you can edit settings to override default values set in resource_registry . This includes values set by OS::Tripleo::Services::ManilaBackendCephFs , which sets defaults for a CephFS back end. 1 ManilaCephFSBackendName sets the name of the manila configuration of your CephFS backend. In this case, the default back end name is cephfs . 2 ManilaCephFSDriverHandlesShareServers controls the lifecycle of the share server. When set to false , the driver does not handle the lifecycle. This is the only supported option. 3 ManilaCephFSCephFSAuthId defines the Ceph auth ID that the director creates for the manila service to access the Ceph cluster. 4 ManilaCephFSCephFSEnableSnapshots controls snapshot activation. The false value indicates that snapshots are not enabled. This feature is currently not supported. For more information about environment files, refer to the Environment Files section in the Director Installation and Usage Guide . 2.2.4. Completing post-deployment configuration You must complete two post-deployment configuration tasks before you create NFS shares, grant user access, and mount NFS shares. Map the neutron StorageNFS network to the isolated data center Storage NFS network. See Section 2.2.4.1, "Configuring the isolated network" Create the default share type. See Section 2.2.4.3, "Configuring a default share type" 2.2.4.1. Configuring the isolated network Map the new isolated StorageNFS network to a neutron-shared provider network. The Compute VMs attach to this neutron network to access share export locations provided by the NFS-Ganesha gateway. For more information about network security with the Shared File Systems service, see Hardening the Shared File System Service in the Security and Hardening Guide . The openstack network create command defines the configuration for the StorageNFS neutron network. You can enter this command with the following options: For --provider-network-type , use the value vlan . For --provider-physical-network , use the default value datacentre , unless you set another tag for the br-isolated bridge through NeutronBridgeMappings in your tripleo-heat-templates. For --provider-segment , use the VLAN value set for the StorageNFS isolated network in the heat template, /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml . This value is 70, unless the deployer modified the isolated network definitions. Procedure On an undercloud node as the stack user, enter the following command: On an undercloud node, enter the openstack network create command to create the StorageNFS network: 2.2.4.2. Configuring the shared provider StorageNFS network Create a corresponding StorageNFSSubnet on the neutron-shared provider network. Ensure that the subnet is the same as the storage_nfs network definition in the network_data.yml file and ensure that the allocation range for the StorageNFS subnet and the corresponding undercloud subnet do not overlap. No gateway is required because the StorageNFS subnet is dedicated to serving NFS shares. Prerequisites The start and ending IP range for the allocation pool. The subnet IP range. 2.2.4.2.1. Configuring the shared provider StorageNFS IPv4 network Procedure Log in to an overcloud node. Source your overcloud credentials. Use the example command to provision the network and make the following updates: Replace the start=172.16.4.150,end=172.16.4.250 IP values with the IP values for your network. Replace the 172.16.4.0/24 subnet range with the subnet range for your network. 2.2.4.2.2. Configuring the shared provider StorageNFS IPv6 network This feature is available in this release as a Technology Preview , and therefore is not fully supported by Red Hat. It should only be used for testing, and should not be deployed in a production environment. For more information about Technology Preview features, see Scope of Coverage Details . Procedure Log in to an overcloud node. Use the sample command to provision the network, updating values as needed. Replace the fd00:fd00:fd00:7000::/64 subnet range with the subnet range for your network. 2.2.4.3. Configuring a default share type You can use the Shared File Systems service to define share types that you can use to create shares with specific settings. Share types work like Block Storage volume types. Each type has associated settings, for example, extra specifications. When you invoke the type during share creation the settings apply to the shared file system. Red Hat OpenStack Platform (RHOSP) director expects a default share type. You must create the default share type before you open the cloud for users to access. For CephFS with NFS, use the manila type-create command: For information about share types, see Creating and managing shares in the Storage Guide .

Commands:
[stack@undercloud-0 ~]USD sudo dnf install -y ceph-ansible [stack@undercloud-0 ~]USD sudo dnf list ceph-ansible Installed Packages ceph-ansible.noarch 3.1.0-0.1.el7
openstack overcloud container image prepare -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/manila.yaml
[stack@undercloud-0 ~]USD grep -E 'ceph|manila' composable_roles/docker-images.yaml DockerCephDaemonImage: 192.168.24.1:8787/rhceph-beta/rhceph-4-rhel8:4-12 DockerManilaApiImage: 192.168.24.1:8787/rhosp-rhel8/openstack-manila-api:2019-01-16 DockerManilaConfigImage: 192.168.24.1:8787/rhosp-rhel8/openstack-manila-api:2019-01-16 DockerManilaSchedulerImage: 192.168.24.1:8787/rhosp-rhel8/openstack-manila-scheduler:2019-01-16 DockerManilaShareImage: 192.168.24.1:8787/rhosp-rhel8/openstack-manila-share:2019-01-16
[stack@undercloud ~]USD cd /usr/share/openstack-tripleo-heat-templates/roles [stack@undercloud roles]USD diff Controller.yaml ControllerStorageNfs.yaml 16a17 > - StorageNFS 50a45 > - OS::TripleO::Services::CephNfs
[stack@undercloud ~]USD openstack overcloud roles generate --roles-path /usr/share/openstack-tripleo-heat-templates/roles -o /home/stack/roles_data.yaml ControllerStorageNfs Compute CephStorage
[stack@undercloud ~]USD openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -n /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml -r /home/stack/roles_data.yaml -e /home/stack/containers-default-parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e/usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-mds.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml
name: StorageNFS enabled: true vip: true name_lower: storage_nfs vlan: 70 ip_subnet: '172.16.4.0/24' allocation_pools: [{'start': '172.16.4.4', 'end': '172.16.4.149'}] ipv6_subnet: 'fd00:fd00:fd00:7000::/64' ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:7000::10', 'end': 'fd00:fd00:fd00:7000:ffff:ffff:ffff:fffe'}]
/usr/share/openstack-tripleo-heat-templates/environments/
[stack@undercloud ~]USD cat /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml A Heat environment file which can be used to enable a a Manila CephFS-NFS driver backend. resource_registry: OS::TripleO::Services::ManilaApi: ../deployment/manila/manila-api-container-puppet.yaml OS::TripleO::Services::ManilaScheduler: ../deployment/manila/manila-scheduler-container-puppet.yaml # Only manila-share is pacemaker managed: OS::TripleO::Services::ManilaShare: ../deployment/manila/manila-share-pacemaker-puppet.yaml OS::TripleO::Services::ManilaBackendCephFs: ../deployment/manila/manila-backend-cephfs.yaml # ceph-nfs (ganesha) service is installed and configured by ceph-ansible # but it's still managed by pacemaker OS::TripleO::Services::CephNfs: ../deployment/ceph-ansible/ceph-nfs.yaml parameter_defaults: ManilaCephFSBackendName: cephfs 1 ManilaCephFSDriverHandlesShareServers: false 2 ManilaCephFSCephFSAuthId: 'manila' 3 ManilaCephFSCephFSEnableSnapshots: false 4 # manila cephfs driver supports either native cephfs backend - 'CEPHFS' # (users mount shares directly from ceph cluster), or nfs-ganesha backend - # 'NFS' (users mount shares through nfs-ganesha server) ManilaCephFSCephFSProtocolHelperType: 'NFS'
[stack@undercloud ~]USD source ~/overcloudrc
(overcloud) [stack@undercloud-0 ~]USD openstack network create StorageNFS --share --provider-network-type vlan --provider-physical-network datacentre --provider-segment 70
[stack@undercloud-0 ~]USD openstack subnet create --allocation-pool start=172.16.4.150,end=172.16.4.250 --dhcp --network StorageNFS --subnet-range 172.16.4.0/24 --gateway none StorageNFSSubnet
[stack@undercloud-0 ~]USD openstack subnet create --ip-version 6 --dhcp --network StorageNFS --subnet-range fd00:fd00:fd00:7000::/64 --gateway none --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful StorageNFSSubnet -f yaml
manila type-create default false