Chapter 7. Deploying storage at the edge You can leverage Red Hat OpenStack Platform director to extend distributed compute node deployments to include distributed image management and persistent storage at the edge with the benefits of using Red Hat OpenStack Platform and Ceph Storage. 7.1. Deploying edge sites with storage After you deploy the central site, build out the edge sites and ensure that each edge location connects primarily to its own storage backend, as well as to the storage back end at the central location. A spine and leaf networking configuration should be included with this configuration, with the addition of the storage and storage_mgmt networks that ceph needs. For more information see Spine leaf networking . You must have connectivity between the storage network at the central location and the storage network at each edge site so that you can move glance images between sites. Ensure that the central location can communicate with the mons and osds at each of the edge sites. However, you should terminate the storage management network at site location boundaries, because the storage management network is used for OSD rebalancing. Procedure Export stack information from the central stack. You must deploy the central stack before running this command: Note The config-download-dir value defaults to /var/lib/mistral/<stack>/ . Create the central_ceph_external.yaml file. This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the steps. When Ceph is deployed without Red Hat OpenStack Platform director, you cannot run the openstack overcloud export ceph command. Manually create the central_ceph_external.yaml file: The fsid parameter is the file system ID of your Ceph Storage cluster: This value is specified in the cluster configuration file in the [global] section: The key parameter is the ceph client key for the openstack account: For more information about the parameters shown in the sample central_ceph_external.yaml file, see Creating a custom environment file . Create the ~/dcn0/glance.yaml file for Image service configuration overrides: Note If you do not use the GlanceRbdPoolName and CephClientUserName parameters for the glance multi-store configuration, then the values are inherited from the parameters that you used to configure the central location. These values might not be the same, and can result in a failed deployment. Configure the ceph.yaml file with configuration parameters relative to the available hardware. For more information, see Mapping the Ceph Storage node disk layout . Implement system tuning by using a file that contains the following parameters tuned to the requirements of you environment: For more information about setting the values for the parameters CephAnsibleExtraConfig , see Setting ceph-ansible group variables . For more information about setting the values for the parameters CephConfigOverrides , see Customizing the Ceph Storage cluster . Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match. The CinderVolumeCluster parameter is included when deploying an edge site with storage. This parameter is used when cinder-volume is deployed as active/active, which is required at edge sites. As a best practice, set the Cinder cluster name to match the availability zone: Generate the roles.yaml file to be used for the dcn0 deployment, for example: Set the number systems in each role by creating the ~/dcn0/roles-counts.yaml file with the desired values for each role. + You must allocate three nodes to satisfy requirements for GlanceApiEdge services. Use the DistributedComputeHCICount parameter for hyperconverged infrastructure. For other architectures, use the DistributedComputeCount parameter. Retrieve the container images for the edge site: Note You must include all environment files to be used for the deployment in the openstack tripleo container image prepare command. Deploy the edge site: Note You must include heat templates for the configuration of networking in your openstack overcloud deploy command. Designing for edge architecture requires spine and leaf networking. See Spine Leaf Networking for more details. You must ensure that nova cell_v2 host mappings are created in the nova API database after the edge locations are deployed. Run the following command on the undercloud: If you scale up an edge site, you must run this command again. 7.2. Deploying edge sites with dedicated Ceph nodes You can deploy dedicated Ceph nodes using Red Hat OpenStack Platform director. Procedure Export stack information from the central stack. You must deploy the central stack before running this command: Note The config-download-dir value defaults to /var/lib/mistral/<stack>/ . Create the central_ceph_external.yaml file. This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the steps. Create the ~/dcn0/glance.yaml file for glance configuration overrides: Configure the ceph.yaml file with configuration parameters relative to the available hardware. For more information, see Mapping the Ceph Storage node disk layout . Configure the naming conventions for your site in the site-name.yaml environment file. The Nova availability zone and the Cinder storage availability zone must match. The CinderVolumeCluster parameter is included when deploying an edge site with storage. This parameter is used when cinder-volume is deployed as active/active, which is required at edge sites. As a best practice, set the Cinder cluster name to match the availability zone: Generate the roles.yaml file to be used for the dcn0 deployment, for example: Set the number systems in each role by creating the ~/dcn0/roles-counts.yaml file with the desired values for each role. You must allocate three nodes for the DistributedCompute role to satisfy requirements for GlanceApiEdge services, and three nodes for the CephAll role. Retrieve the container images for the edge site: Note You must include all environment files to be used for the deployment in the openstack tripleo container image prepare command. Deploy the edge site: Note You must include heat templates for the configuration of networking in your openstack overcloud deploy command. Designing for edge architecture requires spine and leaf networking. See Spine Leaf Networking for more details. You must ensure that nova cell_v2 host mappings are created in the nova API database after the edge locations are deployed. Run the following command on the undercloud: If you scale up an edge site, you must run this command again. 7.3. Using a pre-installed Red Hat Ceph Storage cluster at the edge You can configure Red Hat OpenStack Platform to use a pre-existing Ceph cluster. This is called an external Ceph deployment. Prerequisites You must have a preinstalled Ceph cluster that is local to your DCN site so that latency requirements are not exceeded. Procedure Create the following pools in your Ceph cluster. If you are deploying at the central location, include the backups and metrics pools: Replace <_PGnum_> with the number of placement groups. You can use the Ceph Placement Groups (PGs) per Pool Calculator to determine a suitable value. Create the OpenStack client user in Ceph to provide the Red Hat OpenStack Platform environment access to the appropriate pools: Save the provided Ceph client key that is returned. Use this key as the value for the CephClientKey parameter when you configure the undercloud. Note If you run this command at the central location and plan to use Cinder backup or telemetry services, add allow rwx pool=backups, allow pool=metrics to the command. Save the file system ID of your Ceph Storage cluster. The value of the fsid parameter in the [global] section of your Ceph configuration file is the file system ID: Use this value as the value for the CephClusterFSID parameter when you configure the undercloud. On the undercloud, create an environment file to configure your nodes to connect to the unmanaged Ceph cluster. Use a recognizable naming convention, such as ceph-external-<SITE>.yaml where SITE is the location for your deployment, such as ceph-external-central.yaml, ceph-external-dcn1.yaml, and so on. Use the previously saved values for the CephClusterFSID and CephClientKey parameters. Use a comma delimited list of ip addresses from the Ceph monitors as the value for the CephExternalMonHost parameter. You must select a unique value for the CephClusterName parameter amongst edge sites. Reusing a name will result in the configuration file being overwritten. If you deployed Red Hat Ceph storage using Red Hat OpenStack Platform director at the central location, then you can export the ceph configuration to an environment file central_ceph_external.yaml . This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the steps: If the central location has Red Hat Ceph Storage deployed externally, then you cannot use the openstack overcloud export ceph command to generate the central_ceph_external.yaml file. You must create the central_ceph_external.yaml file manually instead: Create an environment file with similar details about each site with an unmanaged Red Hat Ceph Storage cluster for the central location. The openstack overcloud export ceph command does not work for sites with unmanaged Red Hat Ceph Storage clusters. When you update the central location, this file will allow the central location the storage clusters at your edge sites as secondary locations Use the ceph-ansible-external.yaml, ceph-external-<SITE>.yaml, and the central_ceph_external.yaml environment files when deploying the overcloud: Redeploy the central location after all edge locations have been deployed. 7.4. Creating additional distributed compute node sites A new distributed compute node (DCN) site has its own directory of YAML files on the undercloud. For more information, see Section 4.7, "Managing separate heat stacks" . This procedure contains example commands. Procedure As the stack user on the undercloud, create a new directory for dcn9 : Copy the existing dcn0 templates to the new directory and replace the dcn0 strings with dcn9 : Review the files in the dcn9 directory to confirm that they suit your requirements. Edit undercloud.conf to add a new leaf. In the following example, leaf9 is added to undercloud.conf: Rerun the openstack undercloud install command to update the environment configuration. In your overcloud templates, update the value of the NetworkDeploymentActions parameter from a value of ["CREATE"] , to a value of ["CREATE", "UPDATE"] . If this parameter is not currently included in your templates, add it to one of your environment files, or create a new environment file. Run the deploy script for the central location. Include all templates that you used when you first deployed the central location, as well as the newly created or edited network-environment.yaml file: Verify that your nodes are available and in Provisioning state : When your nodes are available, deploy the new edge site with all appropriate templates: If you've deployed the locations with direct edge-to-edge communication, you must redeploy each edge site to update routes and establish communication with the new location. 7.5. Updating the central location After you configure and deploy all of the edge sites using the sample procedure, update the configuration at the central location so that the central Image service can push images to the edge sites. Warning This procedure restarts the Image service (glance) and interrupts any long running Image service process. For example, if an image is being copied from the central Image service server to a DCN Image service server, that image copy is interrupted and you must restart it. For more information, see Clearing residual data after interrupted Image service processes . Procedure Create a ~/central/glance_update.yaml file similar to the following. This example includes a configuration for two edge sites, dcn0 and dcn1: Create the dcn_ceph.yaml file. In the following example, this file configures the glance service at the central site as a client of the Ceph clusters of the edge sites, dcn0 and dcn1 . Redeploy the central site using the original templates and include the newly created dcn_ceph.yaml and glance_update.yaml files. On a controller at the central location, restart the cinder-volume service. If you deployed the central location with the cinder-backup service, then restart the cinder-backup service too: 7.5.1. Clearing residual data after interrupted Image service processes When you restart the central location, any long-running Image service (glance) processes are interrupted. Before you can restart these processes, you must first clean up residual data on the Controller node that you rebooted, and in the Ceph and Image service databases. Procedure Check and clear residual data in the Controller node that was rebooted. Compare the files in the glance-api.conf file for staging store with the corresponding images in the Image service database, for example <image_ID>.raw . If these corresponding images show importing status, you must recreate the image. If the images show active status, you must delete the data from staging and restart the copy import. Check and clear residual data in Ceph stores. The images that you cleaned from the staging area must have matching records in their stores property in the Ceph stores that contain the image. The image name in Ceph is the image id in the Image service database. Clear the Image service database. Clear any images that are in importing status from the import jobs there were interrupted: 7.6. Deploying Red Hat Ceph Storage Dashboard on DCN Procedure To deploy the Red Hat Ceph Storage Dashboard to the central location, see Adding the Red Hat Ceph Storage Dashboard to an overcloud deployment . These steps should be completed prior to deploying the central location. To deploy Red Hat Ceph Storage Dashboard to edge locations, complete the same steps that you completed for central, however you must complete the following following: Ensure that the ManageNetworks parameter has a value of false in your templates for deploying the edge site. When you set ManageNetworks to false , Edge sites will use the existing networks that were already created in the central stack: You must deploy your own solution for load balancing in order to create a high availability virtual IP. Edge sites do not deploy haproxy, nor pacemaker. When you deploy Red Hat Ceph Storage Dashboard to edge locations, the deployment is exposed on the storage network. The dashboard is installed on each of the three DistributedComputeHCI nodes with distinct IP addresses without a load balancing solution. You can create an additional network to host virtual IP where the Ceph dashboard can be exposed. You must not be reusing network resources for multiple stacks. For more information on reusing network resources, see Reusing network resources in multiple stacks . To create this additional network resource, use the provided network_data_dashboard.yaml heat template. The name of the created network is StorageDashboard . Procedure Log in to Red Hat OpenStack Platform Director as stack . Generate the DistributedComputeHCIDashboard role and any other roles appropriate for your environment: Include the roles.yaml and the network_data_dashboard.yaml in the overcloud deploy command: Note The deployment provides the three ip addresses where the dashboard is enabled on the storage network. Verification To confirm the dashboard is operational at the central location and that the data it displays from the Ceph cluster is correct, see Accessing Ceph Dashboard . You can confirm that the dashboard is operating at an edge location through similar steps, however, there are exceptions as there is no load balancer at edge locations. Retrieve dashboard admin login credentials specific to the selected stack from /var/lib/mistral/<stackname>/ceph-ansible/group_vars/all.yml Within the inventory specific to the selected stack, /var/lib/mistral/<stackname>/ceph-ansible/inventory.yml , locate the DistributedComputeHCI role hosts list and save all three of the storage_ip values. In the example below the first two dashboard IPs are 172.16.11.84 and 172.16.11.87: You can check that the Ceph Dashboard is active at one of these IP addresses if they are accessible to you. These IP addresses are on the storage network and are not routed. If these IP addresses are not available, you must configure a load balancer for the three IP addresses that you get from the inventory to obtain a virtual IP address for verification.

Commands:
openstack overcloud export --config-download-dir /var/lib/mistral/central/ --stack central --output-file ~/dcn-common/central-export.yaml
sudo -E openstack overcloud export ceph --stack central --config-download-dir /var/lib/mistral --output-file ~/dcn-common/central_ceph_external.yaml
parameter_defaults: CephExternalMultiConfig: - cluster: "central" fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33" external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92" keys: - name: "client.openstack" caps: mgr: "allow *" mon: "profile rbd" osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images" key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q==" mode: "0600" dashboard_enabled: false ceph_conf_overrides: client: keyring: /etc/ceph/central.client.openstack.keyring
[global] fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
ceph auth list [client.openstack] key = AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw== caps mgr = "allow *" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics"
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'dcn0 rbd glance store' GlanceBackendID: dcn0 GlanceMultistoreConfig: central: GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' CephClusterName: central GlanceRbdPoolName: images CephClientUserName: openstack
cat > /home/stack/dcn0/ceph.yaml << EOF parameter_defaults: CephClusterName: dcn0 CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb CephPoolDefaultSize: 3 CephPoolDefaultPgNum: 128 EOF
cat > /home/stack/dcn0/tuning.yaml << EOF parameter_defaults: CephAnsibleExtraConfig: is_hci: true CephConfigOverrides: osd_recovery_op_priority: 3 osd_recovery_max_active: 3 osd_max_backfills: 1 ## Set relative to your hardware: # DistributedComputeHCIParameters: # NovaReservedHostMemory: 181000 # DistributedComputeHCIExtraConfig: # nova::cpu_allocation_ratio: 8.2 EOF
cat > /home/stack/central/site-name.yaml << EOF parameter_defaults: NovaComputeAvailabilityZone: dcn0 NovaCrossAZAttach: false CinderStorageAvailabilityZone: dcn0 CinderVolumeCluster: dcn0
openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/roles_data.yaml
parameter_defaults: ControllerCount: 0 ComputeCount: 0 DistributedComputeHCICount: 3 DistributedComputeHCIScaleOutCount: 1 # Optional DistributedComputeScaleOutCount: 1 # Optional
sudo openstack tripleo container image prepare --environment-directory dcn0 -r ~/dcn0/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /home/stack/dcn-common/central-export.yaml -e /home/stack/containers-prepare-parameter.yaml --output-env-file ~/dcn0/dcn0-images-env.yaml
openstack overcloud deploy --stack dcn0 --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/dcn0/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/dnc0/dcn0-images-env.yaml . -e ~/dcn-common/central-export.yaml -e ~/dcn-common/central_ceph_external.yaml -e ~/dcn0/dcn_ceph_keys.yaml -e ~/dcn0/role-counts.yaml -e ~/dcn0/ceph.yaml -e ~/dcn0/site-name.yaml -e ~/dcn0/tuning.yaml -e ~/dcn0/glance.yaml
TRIPLEO_PLAN_NAME=central ansible -i /usr/bin/tripleo-ansible-inventory nova_api[0] -b -a "{{ container_cli }} exec -it nova_api nova-manage cell_v2 discover_hosts --by-service --verbose"
openstack overcloud export --config-download-dir /var/lib/mistral/central/ --stack central --output-file ~/dcn-common/central-export.yaml
sudo -E openstack overcloud export ceph --stack central --config-download-dir /var/lib/mistral --output-file ~/dcn-common/central_ceph_external.yaml
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'dcn0 rbd glance store' GlanceBackendID: dcn0 GlanceMultistoreConfig: central: GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' CephClientUserName: 'openstack' CephClusterName: central
cat > /home/stack/dcn0/ceph.yaml << EOF parameter_defaults: CephClusterName: dcn0 CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb CephPoolDefaultSize: 3 CephPoolDefaultPgNum: 128 EOF
cat > /home/stack/dcn0/site-name.yaml << EOF parameter_defaults: NovaComputeAvailabilityZone: dcn0 NovaCrossAZAttach: false CinderStorageAvailabilityZone: dcn0 CinderVolumeCluster: dcn0
openstack overcloud roles generate DistributedCompute DistributedComputeScaleOut CephAll-o ~/dcn0/roles_data.yaml
parameter_defaults: ControllerCount: 0 ComputeCount: 0 DistributedComputeCount: 3 CephAll: 3 DistributedComputeScaleOutCount: 1 # Optional
sudo openstack tripleo container image prepare --environment-directory dcn0 -r ~/dcn0/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /home/stack/dcn-common/central-export.yaml -e /home/stack/containers-prepare-parameter.yaml --output-env-file ~/dcn0/dcn0-images-env.yaml
openstack overcloud deploy --stack dcn0 --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/dcn0/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-storage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/dnc0/dcn0-images-env.yaml . -e ~/dcn-common/central-export.yaml -e ~/dcn-common/central_ceph_external.yaml -e ~/dcn0/dcn_ceph_keys.yaml -e ~/dcn0/role-counts.yaml -e ~/dcn0/ceph.yaml -e ~/dcn0/site-name.yaml -e ~/dcn0/tuning.yaml -e ~/dcn0/glance.yaml
TRIPLEO_PLAN_NAME=central ansible -i /usr/bin/tripleo-ansible-inventory nova_api[0] -b -a "{{ container_cli }} exec -it nova_api nova-manage cell_v2 discover_hosts --by-service --verbose"
ceph osd pool create volumes <_PGnum_> ceph osd pool create images <_PGnum_> ceph osd pool create vms <_PGnum_> ceph osd pool create backups <_PGnum_> ceph osd pool create metrics <_PGnum_>
ceph auth add client.openstack mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'
[global] fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
parameter_defaults: # The cluster FSID CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19' # The CephX user auth key CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==' # The list of IPs or hostnames of the Ceph monitors CephExternalMonHost: '172.16.1.7, 172.16.1.8, 172.16.1.9' # The desired name of the generated key and conf files CephClusterName: dcn1
sudo -E openstack overcloud export ceph --stack central --config-download-dir /var/lib/mistral --output-file ~/dcn-common/central_ceph_external.yaml
parameter_defaults: CephExternalMultiConfig: - cluster: "central" fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33" external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92" keys: - name: "client.openstack" caps: mgr: "allow *" mon: "profile rbd" osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images" key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q==" mode: "0600" dashboard_enabled: false ceph_conf_overrides: client: keyring: /etc/ceph/central.client.openstack.keyring
parameter_defaults: CephExternalMultiConfig: cluster: dcn1 ... cluster: dcn2 ...
openstack overcloud deploy --stack dcn1 --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/dcn1/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/dnc1/ceph-external-dcn1.yaml . -e ~/dcn-common/central-export.yaml -e ~/dcn-common/central_ceph_external.yaml -e ~/dcn1/dcn_ceph_keys.yaml -e ~/dcn1/role-counts.yaml -e ~/dcn1/ceph.yaml -e ~/dcn1/site-name.yaml -e ~/dcn1/tuning.yaml -e ~/dcn1/glance.yaml
cd ~ mkdir dcn9
cp dcn0/ceph.yaml dcn9/ceph.yaml sed s/dcn0/dcn9/g -i dcn9/ceph.yaml cp dcn0/overrides.yaml dcn9/overrides.yaml sed s/dcn0/dcn9/g -i dcn9/overrides.yaml sed s/"0-ceph-%index%"/"9-ceph-%index%"/g -i dcn9/overrides.yaml cp dcn0/deploy.sh dcn9/deploy.sh sed s/dcn0/dcn9/g -i dcn9/deploy.sh
[leaf0] cidr = 192.168.10.0/24 dhcp_start = 192.168.10.10 dhcp_end = 192.168.10.90 inspection_iprange = 192.168.10.100,192.168.10.190 gateway = 192.168.10.1 masquerade = False ... [leaf9] cidr = 192.168.19.0/24 dhcp_start = 192.168.19.10 dhcp_end = 192.168.19.90 inspection_iprange = 192.168.19.100,192.168.19.190 gateway = 192.168.10.1 masquerade = False
cat > /home/stack/central/network-environment.yaml << EOF parameter_defaults: NetworkDeploymentActions: ["CREATE", "UPDATE"] EOF
openstack overcloud deploy --stack central --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/central/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/central/dcn9-images-env.yaml . -e ~/dcn-common/central-export.yaml -e ~/dcn-common/central_ceph_external.yaml -e ~/central/dcn_ceph_keys.yaml -e ~/central/role-counts.yaml -e ~/central/ceph.yaml -e ~/central/site-name.yaml -e ~/central/tuning.yaml -e ~/central/glance.yaml
openstack baremetal node list
openstack overcloud deploy --stack dcn9 --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/dcn9/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/dnc9/dcn9-images-env.yaml . -e ~/dcn-common/central-export.yaml -e ~/dcn-common/central_ceph_external.yaml -e ~/dcn9/dcn_ceph_keys.yaml -e ~/dcn9/role-counts.yaml -e ~/dcn9/ceph.yaml -e ~/dcn9/site-name.yaml -e ~/dcn9/tuning.yaml -e ~/dcn9/glance.yaml
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' CephClusterName: central GlanceBackendID: central GlanceMultistoreConfig: dcn0: GlanceBackend: rbd GlanceStoreDescription: 'dcn0 rbd glance store' CephClientUserName: 'openstack' CephClusterName: dcn0 GlanceBackendID: dcn0 dcn1: GlanceBackend: rbd GlanceStoreDescription: 'dcn1 rbd glance store' CephClientUserName: 'openstack' CephClusterName: dcn1 GlanceBackendID: dcn1
sudo -E openstack overcloud export ceph --stack dcn0,dcn1 --config-download-dir /var/lib/mistral --output-file ~/central/dcn_ceph.yaml
openstack overcloud deploy --stack central --templates /usr/share/openstack-tripleo-heat-templates/ -r ~/central/central_roles.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml -e ~/central/central-images-env.yaml -e ~/central/role-counts.yaml -e ~/central/site-name.yaml -e ~/central/ceph.yaml -e ~/central/ceph_keys.yaml -e ~/central/glance.yaml -e ~/central/dcn_ceph_external.yaml
ssh heat-admin@controller-0 sudo pcs resource restart openstack-cinder-volume ssh heat-admin@controller-0 sudo pcs resource restart openstack-cinder-backup
glance image-delete <image_id>
parameter_defaults: ManageNetworks: false
openstack overcloud roles generate DistributedComputeHCIDashboard -o ~/dnc0/roles.yaml
openstack overcloud deploy --templates -r ~/<dcn>/<dcn_site_roles>.yaml -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml -e <overcloud_environment_files> -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-dashboard.yaml
DistributedComputeHCI: hosts: dcn1-distributed-compute-hci-0: ansible_host: 192.168.24.16 storage_hostname: dcn1-distributed-compute-hci-0.storage.localdomain storage_ip: 172.16.11.84 dcn1-distributed-compute-hci-1: ansible_host: 192.168.24.22 storage_hostname: dcn1-distributed-compute-hci-1.storage.localdomain storage_ip: 172.16.11.87