Chapter 4. Stretch clusters for Ceph storage As a storage administrator, you can configure stretch clusters by entering stretch mode with 2-site clusters. Red Hat Ceph Storage is capable of withstanding the loss of Ceph OSDs because of its network and cluster, which are equally reliable with failures randomly distributed across the CRUSH map. If a number of OSDs is shut down, the remaining OSDs and monitors still manage to operate. However, this might not be the best solution for some stretched cluster configurations where a significant part of the Ceph cluster can use only a single network component. The example is a single cluster located in multiple data centers, for which the user wants to sustain a loss of a full data center. The standard configuration is with two data centers. Other configurations are in clouds or availability zones. Each site holds two copies of the data, therefore, the replication size is four. The third site should have a tiebreaker monitor, this can be a virtual machine or high-latency compared to the main sites. This monitor chooses one of the sites to restore data if the network connection fails and both data centers remain active. Note The standard Ceph configuration survives many failures of the network or data centers and it never compromises data consistency. If you restore enough Ceph servers following a failure, it recovers. Ceph maintains availability if you lose a data center, but can still form a quorum of monitors and have all the data available with enough copies to satisfy pools' min_size , or CRUSH rules that replicate again to meet the size. Note There are no additional steps to power down a stretch cluster. You can see the Powering down and rebooting Red Hat Ceph Storage cluster for more information. Stretch cluster failures Red Hat Ceph Storage never compromises on data integrity and consistency. If there is a network failure or a loss of nodes and the services can still be restored, Ceph returns to normal functionality on its own. However, there are situations where you lose data availability even if you have enough servers available to meet Ceph's consistency and sizing constraints, or where you unexpectedly do not meet the constraints. First important type of failure is caused by inconsistent networks. If there is a network split, Ceph might be unable to mark OSD as down to remove it from the acting placement group (PG) sets despite the primary OSD being unable to replicate data. When this happens, the I/O is not permitted because Ceph cannot meet its durability guarantees. The second important category of failures is when it appears that you have data replicated across data enters, but the constraints are not sufficient to guarantee this. For example, you might have data centers A and B, and the CRUSH rule targets three copies and places a copy in each data center with a min_size of 2 . The PG might go active with two copies in site A and no copies in site B, which means that if you lose site A, you lose the data and Ceph cannot operate on it. This situation is difficult to avoid with standard CRUSH rules. 4.1. Stretch mode for a storage cluster To configure stretch clusters, you must enter the stretch mode. When stretch mode is enabled, the Ceph OSDs only take PGs as active when they peer across data centers, or whichever other CRUSH bucket type you specified, assuming both are active. Pools increase in size from the default three to four, with two copies on each site. In stretch mode, Ceph OSDs are only allowed to connect to monitors within the same data center. New monitors are not allowed to join the cluster without specified location. If all the OSDs and monitors from a data center become inaccessible at once, the surviving data center will enter a degraded stretch mode. This issues a warning, reduces the min_size to 1 , and allows the cluster to reach an active state with the data from the remaining site. Note The degraded state also triggers warnings that the pools are too small, because the pool size does not get changed. However, a special stretch mode flag prevents the OSDs from creating extra copies in the remaining data center, therefore it still keeps 2 copies. When the missing data center becomes accesible again, the cluster enters recovery stretch mode. This changes the warning and allows peering, but still requires only the OSDs from the data center, which was up the whole time. When all PGs are in a known state and are not degraded or incomplete, the cluster goes back to the regular stretch mode, ends the warning, and restores min_size to its starting value 2 . The cluster again requires both sites to peer, not only the site that stayed up the whole time, therefore you can fail over to the other site, if necessary. Stretch mode limitations It is not possible to exit from stretch mode once it is entered. You cannot use erasure-coded pools with clusters in stretch mode. You can neither enter the stretch mode with erasure-coded pools, nor create an erasure-coded pool when the stretch mode is active. Stretch mode with no more than two sites is supported. The weights of the two sites should be the same. If they are not, you receive the following error: Example To achieve same weights on both sites, the Ceph OSDs deployed in the two sites should be of equal size, that is, storage capacity in the first site is equivalent to storage capacity in the second site. While it is not enforced, you should run two Ceph monitors on each site and a tiebreaker, for a total of five. This is because OSDs can only connect to monitors in their own site when in stretch mode. You have to create your own CRUSH rule, which provides two copies on each site, which totals to four on both sites. You cannot enable stretch mode if you have existing pools with non-default size or min_size . Because the cluster runs with min_size 1 when degraded, you should only use stretch mode with all-flash OSDs. This minimizes the time needed to recover once connectivity is restored, and minimizes the potential for data loss. Additional Resources See Troubleshooting clusters in stretch mode for troubleshooting steps. 4.1.1. Setting the crush location for the daemons Before you enter the stretch mode, you need to prepare the cluster by setting the crush location to the daemons in the Red Hat Ceph Storage cluster. There are two ways to do this: Bootstrap the cluster through a service configuration file, where the locations are added to the hosts as part of deployment. Set the locations manually through ceph osd crush add-bucket and ceph osd crush move commands after the cluster is deployed. Method 1: Bootstrapping the cluster Prerequisites Root-level access to the nodes. Procedure If you are bootstrapping your new storage cluster, you can create the service configuration .yaml file that adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services should run: Example Bootstrap the storage cluster with the --apply-spec option: Syntax Example Important You can use different command options with the cephadm bootstrap command. However, always include the --apply-spec option to use the service configuration file and configure the host locations. Additional Resources See Bootstrapping a new storage cluster for more information about Ceph bootstrapping and different cephadm bootstrap command options. Method 2: Setting the locations after the deployment Prerequisites Root-level access to the nodes. Procedure Add two buckets to which you plan to set the location of your non-tiebreaker monitors to the CRUSH map, specifying the bucket type as as datacenter : Syntax Example Move the buckets under root=default : Syntax Example Move the OSD hosts according to the required CRUSH placement: Syntax Example 4.1.2. Entering the stretch mode The new stretch mode is designed to handle two sites. There is a lower risk of component availability outages with 2-site clusters. Prerequisites Root-level access to the nodes. The crush location is set to the hosts. Procedure Set the location of each monitor, matching your CRUSH map: Syntax Example Generate a CRUSH rule which places two copies on each data center: Syntax Example Edit the decompiled CRUSH map file to add a new rule: Example 1 The rule id has to be unique. In this example, there is only one more rule with id 0 , thereby the id 1 is used, however you might need to use a different rule ID depending on the number of existing rules. 2 3 In this example, there are two data center buckets named DC1 and DC2 . Note This rule makes the cluster have read-affinity towards data center DC1 . Therefore, all the reads or writes happen through Ceph OSDs placed in DC1 . If this is not desirable, and reads or writes are to be distributed evenly across the zones, the crush rule is the following: Example In this rule, the data center is selected randomly and automatically. See CRUSH rules for more information on firstn and indep options. Inject the CRUSH map to make the rule available to the cluster: Syntax Example If you do not run the monitors in connectivity mode, set the election strategy to connectivity : Example Enter stretch mode by setting the location of the tiebreaker monitor to split across the data centers: Syntax Example In this example the monitor mon.host07 is the tiebreaker. Important The location of the tiebreaker monitor should differ from the data centers to which you previously set the non-tiebreaker monitors. In the example above, it is data center DC3 . Important Do not add this data center to the CRUSH map as it results in the following error when you try to enter stretch mode: Note If you are writing your own tooling for deploying Ceph, you can use a new --set-crush-location option when booting monitors, instead of running the ceph mon set_location command. This option accepts only a single bucket=location pair, for example ceph-mon --set-crush-location 'datacenter=DC1' , which must match the bucket type you specified when running the enable_stretch_mode command. Verify that the stretch mode is enabled successfully: Example The stretch_mode_enabled should be set to true . You can also see the number of stretch buckets, stretch mode buckets, and if the stretch mode is degraded or recovering. Verify that the monitors are in an appropriate locations: Example You can also see which monitor is the tiebreaker, and the monitor election strategy. Additional Resources See Configuring monitor election strategy for more information about monitor election strategy. 4.1.3. Adding OSD hosts in stretch mode You can add Ceph OSDs in the stretch mode. The procedure is similar to the addition of the OSD hosts on a cluster where stretch mode is not enabled. Prerequisites A running Red Hat Ceph Storage cluster. Stretch mode in enabled on a cluster. Root-level access to the nodes. Procedure List the available devices to deploy OSDs: Syntax Example Deploy the OSDs on specific hosts or on all the available devices: Create an OSD from a specific device on a specific host: Syntax Example Deploy OSDs on any available and unused devices: Important This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command. Example Move the OSD hosts under the CRUSH bucket: Syntax Example Note Ensure you add the same topology nodes on both sites. Issues might arise if hosts are added only on one site. Additional Resources See Adding OSDs for more information about the addition of Ceph OSDs.

Commands:
ceph mon enable_stretch_mode host05 stretch_rule datacenter Error EINVAL: the 2 datacenter instances in the cluster have differing weights 25947 and 15728 but stretch mode currently requires they be the same!
service_type: host addr: host01 hostname: host01 location: root: default datacenter: DC1 labels: - osd - mon - mgr --- service_type: host addr: host02 hostname: host02 location: datacenter: DC1 labels: - osd - mon --- service_type: host addr: host03 hostname: host03 location: datacenter: DC1 labels: - osd - mds - rgw --- service_type: host addr: host04 hostname: host04 location: root: default datacenter: DC2 labels: - osd - mon - mgr --- service_type: host addr: host05 hostname: host05 location: datacenter: DC2 labels: - osd - mon --- service_type: host addr: host06 hostname: host06 location: datacenter: DC2 labels: - osd - mds - rgw --- service_type: host addr: host07 hostname: host07 labels: - mon --- service_type: mon placement: label: "mon" --- service_id: cephfs placement: label: "mds" --- service_type: mgr service_name: mgr placement: label: "mgr" --- service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: label: "osd" spec: data_devices: all: true --- service_type: rgw service_id: objectgw service_name: rgw.objectgw placement: count: 2 label: "rgw" spec: rgw_frontend_port: 8080
cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD
cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1
ceph osd crush add-bucket BUCKET_NAME BUCKET_TYPE
ceph osd crush add-bucket DC1 datacenter ceph osd crush add-bucket DC2 datacenter
ceph osd crush move BUCKET_NAME root=default
ceph osd crush move DC1 root=default ceph osd crush move DC2 root=default
ceph osd crush move HOST datacenter= DATACENTER
ceph osd crush move host01 datacenter=DC1
ceph mon set_location HOST datacenter= DATACENTER
ceph mon set_location host01 datacenter=DC1 ceph mon set_location host02 datacenter=DC1 ceph mon set_location host04 datacenter=DC2 ceph mon set_location host05 datacenter=DC2 ceph mon set_location host07 datacenter=DC3
ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME
ceph osd getcrushmap > crush.map.bin crushtool -d crush.map.bin -o crush.map.txt
rule stretch_rule { id 1 1 type replicated min_size 1 max_size 10 step take DC1 2 step chooseleaf firstn 2 type host step emit step take DC2 3 step chooseleaf firstn 2 type host step emit }
rule stretch_rule { id 1 type replicated min_size 1 max_size 10 step take default step choose firstn 0 type datacenter step chooseleaf firstn 2 type host step emit }
crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME
crushtool -c crush.map.txt -o crush2.map.bin ceph osd setcrushmap -i crush2.map.bin
ceph mon set election_strategy connectivity
ceph mon set_location HOST datacenter= DATACENTER ceph mon enable_stretch_mode HOST stretch_rule datacenter
ceph mon set_location host07 datacenter=DC3 ceph mon enable_stretch_mode host07 stretch_rule datacenter
Error EINVAL: there are 3 datacenters in the cluster but stretch mode currently only works with 2!
ceph osd dump epoch 361 fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d created 2023-01-16T05:47:28.482717+0000 modified 2023-01-17T17:36:50.066183+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 31 full_ratio 0.95 backfillfull_ratio 0.92 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release quincy stretch_mode_enabled true stretch_bucket_count 2 degraded_stretch_mode 0 recovering_stretch_mode 0 stretch_mode_bucket 8
ceph mon dump epoch 19 fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d last_changed 2023-01-17T04:12:05.709475+0000 created 2023-01-16T05:47:25.631684+0000 min_mon_release 16 (pacific) election_strategy: 3 stretch_mode_enabled 1 tiebreaker_mon host07 disallowed_leaders host07 0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host07; crush_location {datacenter=DC3} 1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2} 2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1} 3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1} 4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2} dumped monmap epoch 19
ceph orch device ls [--hostname= HOST_1 HOST_2 ] [--wide] [--refresh]
ceph orch device ls
ceph orch daemon add osd HOST : DEVICE_PATH
ceph orch daemon add osd host03:/dev/sdb
ceph orch apply osd --all-available-devices
ceph osd crush move HOST datacenter= DATACENTER
ceph osd crush move host03 datacenter=DC1 ceph osd crush move host06 datacenter=DC2