Chapter 5. Generalized stretch cluster configuration for three availability zones As a storage administrator, you can configure a generalized stretch cluster configuration for three availability zones with Ceph OSDs. Ceph can withstand the loss of Ceph OSDs because of its network and cluster, which are equally reliable with failures randomly distributed across the CRUSH map. If a number of OSDs are shut down, the remaining OSDs and monitors still manage to operate. Using a single cluster limits data availability to a single location with a single point of failure. However, in some situations, higher availability might be required. Using three availability zones allows the cluster to withstand power loss and even a full data center loss in the event of a natural disaster. With a generalized stretch cluster configuration for three availability zones, three data centers are supported, with each site holding two copies of the data. This helps ensure that even during a data center outage, the data remains accessible and writeable from another site. With this configuration, the pool replication size is 6 and the pool min_size is 3 . Note The standard Ceph configuration survives many failures of the network or data centers and it never compromises data consistency. If you restore enough Ceph servers following a failure, it recovers. Ceph maintains availability if you lose a data center, but can still form a quorum of monitors and have all the data available with enough copies to satisfy pools' min_size , or CRUSH rules that replicate again to meet the size. 5.1. Generalized stretch cluster deployment limitations When using generalized stretch clusters, the following limitations should be considered. Generalized stretch cluster configuration for three availability zones does not support I/O operations during a netsplit scenario between two or more zones. While the cluster remains accessible for basic Ceph commands, I/O usage remains unavailable until the netsplit is resolved. This is different from stretch mode, where the tiebreaker monitor can isolate one zone of the cluster and continue I/O operations in degraded mode during a netsplit. For more information about stretch mode, see Stretch mode for a storage cluster . In a three availability zone configuration, Red Hat Ceph Storage is designed to tolerate multiple host failures. However, if more than 25% of the OSDs in the cluster go down, Ceph may stop marking OSDs as out . This behavior is controlled by the mon_osd_min_in_ratio parameter. By default, mon_osd_min_in_ratio is set to 0.75, meaning that at least 75% of the OSDs in the cluster must remain in (active) before any additional OSDs can be marked out . This setting prevents too many OSDs from being marked out as this might lead to significant data movement. The data movement can cause high client I/O impact and long recovery times when the OSDs are returned to service. If Red Hat Ceph Storage stops marking OSDs as out, some placement groups (PGs) may fail to rebalance to surviving OSDs, potentially leading to inactive placement groups (PGs). Important While adjusting the mon_osd_min_in_ratio value can allow more OSDs to be marked out and trigger rebalancing, this should be done with caution. For more information on the mon_osd_min_in_ratio parameter, see Ceph Monitor and OSD configuration options . 5.2. Generalized stretch cluster deployment requirements This information details important hardware, software, and network requirements that are needed for deploying a generalized stretch cluster configuration for three availability zones. 5.2.1. Hardware requirements Use the following minimum hardware requirements before deploying generalized stretch cluster configuration for three availability zones. The following table lists the physical server locations and Ceph component layout for an example three availability zone deployment. Table 5.1. Hardware requirements Host name Datacenter Ceph services host01 DC1 OSD+MON+MGR host02 DC1 OSD+MON+MGR+RGW host03 DC1 OSD+MON+MDS host04 DC2 OSD+MON+MGR host05 DC2 OSD+MON+MGR+RGW host06 DC2 OSD+MON+MDS host07 DC3 OSD+MON+MGR host08 DC3 OSD+MON+MGR+RGW host09 DC3 OSD+MON+MDS 5.2.2. Network configuration requirements Use the following network configuration requirements before deploying generalized stretch cluster configuration for three availability zones. Have two separate networks, one public network and one cluster network. Have three different data centers that support VLANS and subnets for Ceph cluster and public networks for all data centers. Note You can use different subnets for each of the data centers. The latencies between data centers running the Red Hat Ceph Storage Object Storage Devices (OSDs) cannot exceed 10 ms RTT. For more information about network considerations, see Network considerations for Red Hat Ceph Storage in the Red Hat Ceph Storage Hardware Guide. 5.2.3. Cluster setup requirements Ensure that the hostname is configured by using the bare or short hostname in all hosts. Syntax Note The hostname command should only return the short hostname, when run on all nodes. If the FQDN is returned, the cluster configuration will not be successful. 5.3. Bootstrapping the Ceph cluster with a specification file Deploy the generalized stretch cluster by setting the CRUSH location to the daemons in the cluster with the spec configuration file. Set the CRUSH location to the daemons in the cluster with a service configuration file. Use the configuration file to add the hosts to the proper locations during deployment. For more information about Ceph bootstrapping and different cephadm bootstrap command options, see Bootstrapping a new storage cluster in the Red Hat Ceph Storage Installation Guide. Important Run cephadm bootstrap on the node that you want to be the initial Monitor node in the cluster. The IP_ADDRESS option should be the IP address of the node you are using to run cephadm bootstrap . Note If the storage cluster includes multiple networks and interfaces, be sure to choose a network that is accessible by any node that uses the storage cluster. To deploy a storage cluster by using IPV6 addresses, use the IPV6 address format for the --mon-ip <IP_ADDRESS> option. For example: cephadm bootstrap --mon-ip 2620:52:0:880:225:90ff:fefc:2536 --registry-json /etc/mylogin.json . To route the internal cluster traffic over the public network, omit the --cluster-network SUBNET option. Within this procedure the network Classless Inter-Domain Routing (CIDR) is referred to as subnet . Prerequisites Be sure that you have root-level access to the nodes. Procedure Create the service configuration YAML file. The YAML file adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services run. The following example depends on the specific OSD and Ceph Object Gateway (RGW) configuration that is needed. Syntax For more information about changing the custom spec for OSD and Object Gateway, see the following deployment instructions: * Deploying Ceph OSDs using advanced service specifications in the Red Hat Ceph Storage Operations Guide. * Deploying the Ceph Object Gateway using the service specification in the Red Hat Ceph Storage Object Gateway Guide. Bootstrap the storage cluster with the --apply-spec option. Syntax Example Important You can use different command options with the cephadm bootstrap command but always include the --apply-spec option to use the service configuration file and configure the host locations. Log into the cephadm shell. Syntax Example Configure the public network with the subnet. For more information about configuring multiple public networks to the cluster, see Configuring multiple public networks to the cluster in the Red Hat Ceph Storage Configuration Guide. Syntax Example Optional: Configure a cluster network. For more information about configuring multiple cluster networks to the cluster, see Configuring a private network in the Red Hat Ceph Storage Configuration Guide. Syntax Example Optional: Verify the network configurations. Syntax Example Restart the daemons. Ceph daemons bind dynamically, so you do not have to restart the entire cluster at once if you change the network configuration for a specific daemon. Syntax Optional: To restart the cluster on the admin node as a root user, run the systemctl restart command. Note To get the FSID of the cluster, use the ceph fsid command. Syntax Example Verification Verify the specification file details and that the bootstrap was installed successfully. Verify that all hosts were placed in the expected data centers, as specified in step 1 of the procedure. Syntax Check that there are three data centers under root and that the hosts are placed in each of the expected data centers. Note The hosts with OSDs will only be present after bootstrap if OSDs are deployed during bootstrap with the specification file. Example From the cephadm shell, verify that the mon daemons are deployed with CRUSH locations, as specified in step 1 of the procedure. Syntax + Check that all mon daemons are in the output and that the correct CRUSH locations are added. + .Example --- [root@host01 ~]# ceph mon dump epoch 19 fsid b556497a-693a-11ef-b9d1-fa163e841fd7 last_changed 2024-09-03T12:47:08.419495+0000 created 2024-09-02T14:50:51.490781+0000 min_mon_release 19 (squid) election_strategy: 3 0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1} 1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1} 2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1} 3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2} 4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3} 5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2} 6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3} 7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3} 8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2} dumped monmap epoch 19 --- Verify that the service spec and all location attributes are added correctly. Check the service name for mon daemons on the cluster, by using the ceph orch ls command. Example Confirm the mon daemon services, by using the ceph orch ls mon --export command. Example Verify that the bootstrap was installed successfully, by running the cephadm shell ceph -s command. For more information, see Verifying the cluster installation. 5.4. Enabling three availability zones on the pool Use this information to enable and integrate three availability zones within a generalized stretch cluster configuration. Prerequisites Before you begin, make sure that you have the following prerequisites in place: * Root-level access to the nodes. * The CRUSH location is set to the hosts. Procedure Get the most recent CRUSH map and decompile the map into a text file. Syntax Example Add the new CRUSH rule into the decompiled CRUSH map file from the . In this example, the rule name is 3az_rule . Syntax With this rule, the placement groups will be replicated with two copies in each of the three data centers. Inject the CRUSH map to make the rule available to the cluster. Syntax Example You can verify that the rule was injected successfully, by using the following steps. List the rules on the cluster. Syntax Example Dump the CRUSH rule. Syntax Example Set the MON election strategy to connectivity. Syntax When updated successfully, the election_strategy is updated to 3 . The default election_strategy is 1 . Optional: Verify the election strategy that was set in the step. Syntax Check that all mon daemons are in the output and that the correct CRUSH locations are added. Example Set the pool to associate with three availability zone stretch clusters. For more information about available pool values, see Pool values in the Red Hat Ceph Storage Storage Strategies Guide. Syntax Replace the variables as follows: POOL_NAME The name of the pool. It must be an existing pool, this command doesn't create a new pool. PEERING_CRUSH_BUCKET_COUNT The value is used along with peering_crush_bucket_barrier to determined whether the set of OSDs in the chosen acting set can peer with each other, based on the number of distinct buckets there are in the acting set. PEERING_CRUSH_BUCKET_TARGET This value is used along with peering_crush_bucket_barrier and size to calculate the value bucket_max which limits the number of OSDs in the same bucket from getting chose to be in the acting set of a PG. PEERING_CRUSH_BUCKET_BARRIER The type of bucket a pool is stretched across. For example, rack, row, or datacenter. CRUSH_RULE The crush rule to use for the stretch pool. The type of pool must match the type of crush_rule (replicated or erasure). SIZE The number of replicas for objects in the stretch pool. MIN_SIZE The minimum number of replicas required for I/O in the stretch pool. Important The `--yes-i-really-mean-it flag is required when setting the PEERING_CRUSH_BUCKET_COUNT and PEERING_CRUSH_BUCKET_TARGET to be more than the number of buckets in the CRUSH map. Use the optional flag to confirm that you want to bypass the safety checks and set the values for a stretch pool. Example Note To revert a pool to a nonstretched cluster, use the ceph osd pool stretch unset POOL_NAME command. Using this command does not unset the crush_rule , size , and min_size values. If needed, these need to be reset manually. A success message is emitted that the pool stretch values were set correctly. Optional: Verify the pools associated with the stretch clusters, by using the ceph osd pool stretch show commands. Example 5.5. Adding OSD hosts with three availability zones You can add Ceph OSDs with three availability zones on a generalized stretch cluster. The procedure is similar to the addition of the OSD hosts on a cluster where a generalized stretch cluster is not enabled. For more information, see Adding OSDs in the Red Hat Ceph Storage Installing Guide. Prerequisites Before you begin, make sure that you have the following prerequisites in place: * A running Red Hat Ceph Storage cluster. * Three availability zones enabled on a cluster. For more information, see _Enabling three availability zones on the pool . * Root-level access to the nodes. Procedure From the node that contains the admin keyring, install the storage cluster's public SSH key in the root user's authorized_keys file on the new host. Syntax Example Optional: Verify the status of the storage cluster and that each new host has been added by using the ceph orch host ls command. See that the new host has been added and that the Status of each host is blank in the output. List the available devices to deploy OSDs. Deploy in one of the following ways: Create an OSD from a specific device on a specific host. Syntax Example Deploy OSDs on any available and unused devices. Important This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command. Syntax Move the OSD hosts under the CRUSH bucket. Syntax Example Note Ensure you add the same topology nodes on all sites. Issues might arise if hosts are added only on one site. Verification Verify that all hosts are moved to the assigned data centers, by using the ceph osd tree command.

Commands:
hostnamectl set-hostname SHORT_NAME
service_type: host hostname: HOST01 addr: IP_ADDRESS01 labels: ['alertmanager', 'osd', 'installer', '_admin', 'mon', 'prometheus', 'mgr', 'grafana'] location: root: default datacenter: DC1 --- service_type: host hostname: HOST02 addr: IP_ADDRESS02 labels: ['osd', 'mon', 'mgr', 'rgw'] location: root: default datacenter: DC1 --- service_type: host hostname: HOST03 addr: IP_ADDRESS03 labels: ['osd', 'mon', 'mds'] location: root: default datacenter: DC1 --- service_type: host hostname: HOST04 addr: IP_ADDRESS04 labels: ['osd', '_admin', 'mon', 'mgr'] location: root: default datacenter: DC2 --- service_type: host hostname: HOST05 addr: IP_ADDRESS05 labels: ['osd', 'mon', 'mgr', 'rgw'] location: root: default datacenter: DC2 --- service_type: host hostname: HOST06 addr: IP_ADDRESS06 labels: ['osd', 'mon', 'mds'] location: root: default datacenter: DC2 --- service_type: host hostname: HOST07 addr: IP_ADDRESS07 labels: ['osd', '_admin', 'mon', 'mgr'] location: root: default datacenter: DC3 --- service_type: host hostname: HOST08 addr: IP_ADDRESS08 labels: ['osd', 'mon', 'mgr', 'rgw'] location: root: default datacenter: DC3 --- service_type: host hostname: HOST09 addr: IP_ADDRESS09 labels: ['osd', 'mon', 'mds'] location: root: default datacenter: DC3 --- service_type: mon service_name: mon placement: label: mon spec: crush_locations: HOST01: - datacenter=DC1 HOST02: - datacenter=DC1 HOST03: - datacenter=DC1 HOST04: - datacenter=DC2 HOST05: - datacenter=DC2 HOST06: - datacenter=DC2 HOST07: - datacenter=DC3 HOST08: - datacenter=DC3 HOST09: - datacenter=DC3 --- service_type: mgr service_name: mgr placement: label: mgr ------ service_type: osd service_id: osds placement: label: osd spec: data_devices: all: true --------- service_type: rgw service_id: rgw.rgw.1 placement: label: rgw ------------
cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD
cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1
cephadm shell
cephadm shell
ceph config set global public_network " SUBNET_1 , SUBNET_2 , ..."
ceph config global mon public_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"
ceph config set global cluster_network " SUBNET_1 , SUBNET_2 , ..."
ceph config set global cluster_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"
ceph config dump | grep network
ceph config dump | grep network
ceph orch restart mon
systemctl restart ceph- FSID_OF_CLUSTER .target
systemctl restart ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad.target
ceph osd tree
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.87836 root default -3 0.29279 datacenter DC1 -2 0.09760 host host01-installer 0 hdd 0.02440 osd.0 up 1.00000 1.00000 12 hdd 0.02440 osd.12 up 1.00000 1.00000 21 hdd 0.02440 osd.21 up 1.00000 1.00000 29 hdd 0.02440 osd.29 up 1.00000 1.00000 -4 0.09760 host host02 1 hdd 0.02440 osd.1 up 1.00000 1.00000 9 hdd 0.02440 osd.9 up 1.00000 1.00000 18 hdd 0.02440 osd.18 up 1.00000 1.00000 28 hdd 0.02440 osd.28 up 1.00000 1.00000 -5 0.09760 host host03 8 hdd 0.02440 osd.8 up 1.00000 1.00000 16 hdd 0.02440 osd.16 up 1.00000 1.00000 24 hdd 0.02440 osd.24 up 1.00000 1.00000 34 hdd 0.02440 osd.34 up 1.00000 1.00000 -7 0.29279 datacenter DC2 -6 0.09760 host host04 4 hdd 0.02440 osd.4 up 1.00000 1.00000 13 hdd 0.02440 osd.13 up 1.00000 1.00000 20 hdd 0.02440 osd.20 up 1.00000 1.00000 27 hdd 0.02440 osd.27 up 1.00000 1.00000 -8 0.09760 host host05 3 hdd 0.02440 osd.3 up 1.00000 1.00000 10 hdd 0.02440 osd.10 up 1.00000 1.00000 19 hdd 0.02440 osd.19 up 1.00000 1.00000 30 hdd 0.02440 osd.30 up 1.00000 1.00000 -9 0.09760 host host06 7 hdd 0.02440 osd.7 up 1.00000 1.00000 17 hdd 0.02440 osd.17 up 1.00000 1.00000 26 hdd 0.02440 osd.26 up 1.00000 1.00000 35 hdd 0.02440 osd.35 up 1.00000 1.00000 -11 0.29279 datacenter DC3 -10 0.09760 host host07 5 hdd 0.02440 osd.5 up 1.00000 1.00000 14 hdd 0.02440 osd.14 up 1.00000 1.00000 23 hdd 0.02440 osd.23 up 1.00000 1.00000 32 hdd 0.02440 osd.32 up 1.00000 1.00000 -12 0.09760 host host08 2 hdd 0.02440 osd.2 up 1.00000 1.00000 11 hdd 0.02440 osd.11 up 1.00000 1.00000 22 hdd 0.02440 osd.22 up 1.00000 1.00000 31 hdd 0.02440 osd.31 up 1.00000 1.00000 -13 0.09760 host host09 6 hdd 0.02440 osd.6 up 1.00000 1.00000 15 hdd 0.02440 osd.15 up 1.00000 1.00000 25 hdd 0.02440 osd.25 up 1.00000 1.00000 33 hdd 0.02440 osd.33 up 1.00000 1.00000
ceph mon dump
ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 8m ago 6d count:1 ceph-exporter 9/9 8m ago 6d * crash 9/9 8m ago 6d * grafana ?:3000 1/1 8m ago 6d count:1 mds.cephfs 3/3 8m ago 6d label:mds mgr 6/6 8m ago 6d label:mgr mon 9/9 8m ago 5d label:mon node-exporter ?:9100 9/9 8m ago 6d * osd.all-available-devices 36 8m ago 6d label:osd prometheus ?:9095 1/1 8m ago 6d count:1 rgw.rgw.1 ?:80 3/3 8m ago 6d label:rgw
ceph orch ls mon --export service_type: mon service_name: mon placement: label: mon spec: crush_locations: host01-installer: - datacenter=DC1 host02: - datacenter=DC1 host03: - datacenter=DC1 host04: - datacenter=DC2 host05: - datacenter=DC2 host06: - datacenter=DC2 host07: - datacenter=DC3 host08: - datacenter=DC3 host09: - datacenter=DC3
ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME
ceph osd getcrushmap > crush.map.bin crushtool -d crush.map.bin -o crush.map.txt
rule 3az_rule { id 1 type replicated step take default step choose firstn 3 type datacenter step chooseleaf firstn 2 type host step emit }
crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME
crushtool -c crush.map.txt -o crush2.map.bin ceph osd setcrushmap -i crush2.map.bin
ceph osd crush rule ls
ceph osd crush rule ls replicated_rule ec86_pool 3az_rule
ceph osd crush rule dump CRUSH_RULE
ceph osd crush rule dump 3az_rule { "rule_id": 1, "rule_name": "3az_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_firstn", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_firstn", "num": 2, "type": "host" }, { "op": "emit" } ] }
ceph mon set election_strategy connectivity
ceph mon dump
ceph mon dump epoch 19 fsid b556497a-693a-11ef-b9d1-fa163e841fd7 last_changed 2024-09-03T12:47:08.419495+0000 created 2024-09-02T14:50:51.490781+0000 min_mon_release 19 (squid) election_strategy: 3 0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1} 1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1} 2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1} 3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2} 4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3} 5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2} 6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3} 7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3} 8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2} dumped monmap epoch 19
ceph osd pool stretch set _POOL_NAME_ _PEERING_CRUSH_BUCKET_COUNT_ _PEERING_CRUSH_BUCKET_TARGET_ _PEERING_CRUSH_BUCKET_BARRIER_ _CRUSH_RULE_ _SIZE_ _MIN_SIZE_ [--yes-i-really-mean-it]
ceph osd pool stretch set pool01 2 3 datacenter 3az_rule 6 3
ceph osd pool stretch show pool01 pool: pool01 pool_id: 1 is_stretch_pool: 1 peering_crush_bucket_count: 2 peering_crush_bucket_target: 3 peering_crush_bucket_barrier: 8 crush_rule: 3az_rule size: 6 min_size: 3
ssh-copy-id -f -i /etc/ceph/ceph.pub user@ NEWHOST
ssh-copy-id -f -i /etc/ceph/ceph.pub root@host11 ssh-copy-id -f -i /etc/ceph/ceph.pub root@host12
ceph orch daemon add osd _HOST_:_DEVICE_PATH_
ceph orch daemon add osd host11:/dev/sdb
ceph orch apply osd --all-available-devices
ceph osd crush move HOST datacenter= DATACENTER
ceph osd crush move host10 datacenter=DC1 ceph osd crush move host11 datacenter=DC2 ceph osd crush move host12 datacenter=DC3