Chapter 3. Configuring multisite storage replication Mirroring or replication is enabled on a per CephBlockPool basis within peer managed clusters and can then be configured on a specific subset of images within the pool. The rbd-mirror daemon is responsible for replicating image updates from the local peer cluster to the same image in the remote cluster. These instructions detail how to create the mirroring relationship between two OpenShift Data Foundation managed clusters. 3.1. Enabling OMAP generator and volume replication on managed clusters Execute the following steps on the Primary managed cluster and the Secondary managed cluster to enable the OMAP and Volume-Replication CSI sidecar containers in the csi-rbdplugin-provisioner Pods. Procedure Run the following patch command to set the value to true for CSI_ENABLE_OMAP_GENERATOR in the rook-ceph-operator-config ConfigMap. Example output: Run the following patch command to set the value to true for CSI_ENABLE_VOLUME_REPLICATION in the rook-ceph-operator-config ConfigMap. Example output: Validate that the following two new CSI sidecar containers per csi-rbdplugin-provisioner pod are added. Example output: Note The new containers are repeated because there are two csi-rbdplugin-provisioner pods for redundancy. 3.2. Installing OpenShift Data Foundation Multicluster Orchestrator OpenShift Data Foundation Multicluster Orchestrator is a controller that is installed from OpenShift Container Platform's OperatorHub on the Hub cluster. This Multicluster Orchestrator controller, along with the MirrorPeer custom resource, creates a bootstrap token and exchanges this token between the managed clusters. Procedure Navigate to OperatorHub on the Hub cluster and use the keyword filter to search for ODF Multicluster Orchestrator . Click ODF Multicluster Orchestrator tile. Keep all default settings and click Install. The operator resources are installed in openshift-operators and available to all namespaces. Verify that the ODF Multicluster Orchestrator shows a green tick indicating successful installation. 3.3. Creating mirror peer on hub cluster Mirror Peer is a cluster-scoped resource to hold information about the managed clusters that will have a peer-to-peer relationship. Prerequisites Ensure that ODF Multicluster Orchestrator is installed on the Hub cluster . You must have only two clusters per Mirror Peer. Ensure that each cluster has uniquely identifiable cluster names such as ocp4perf1 and ocp4perf2 . Procedure Click ODF Multicluster Orchestrator to view the operator details. You can also click View Operator after the Multicluster Orchestrator is installed successfully. Click on Mirror Peer API Create instance and then select YAML view. Create Mirror Peer in YAML view. Copy the following YAML to filename mirror-peer.yaml after replacing <cluster1> and <cluster2> with the correct names of your managed clusters in the RHACM console. Note There is no need to specify a namespace to create this resource because MirrorPeer is a cluster-scoped resource. Copy the contents of your unique mirror-peer.yaml file into the YAML view. You must completely replace the original content. Click Create at the bottom of the YAML view screen. Verify that you can view Phase status as ExchangedSecret . Note In some deployments, the output for the validation can also be ExchangingSecret which is also an acceptable result. 3.4. Enabling Mirroring on Managed clusters To enable mirroring, you must enable the mirroring setting of the storage cluster for each managed cluster. This is a manual step using CLI and the oc patch command. Important You must run the oc patch storagecluster command on the Primary managed cluster and the Secondary managed cluster as well as the follow-on validation commands after the StorageCluster has mirroring enabled. Procedure Enable cluster level mirroring flag using storage cluster name. Example output: Validate that mirroring is enabled on the default Ceph block pool. Example output: Validate that the rbd-mirror pod is up and running. Example output: Validate the status of the daemon health. Example output: Note It could take up to 10 minutes for the daemon health and health fields to change from Warning to OK . If the status does not change to OK in approximately 10 minutes then use the RHACM console to verify that the Submariner add-on connection is still in a Healthy state.

Commands:
oc patch cm rook-ceph-operator-config -n openshift-storage --type json --patch '[{ "op": "add", "path": "/data/CSI_ENABLE_OMAP_GENERATOR", "value": "true" }]'
configmap/rook-ceph-operator-config patched
oc patch cm rook-ceph-operator-config -n openshift-storage --type json --patch '[{ "op": "add", "path": "/data/CSI_ENABLE_VOLUME_REPLICATION", "value": "true" }]'
configmap/rook-ceph-operator-config patched
for l in USD(oc get pods -n openshift-storage -l app=csi-rbdplugin-provisioner -o jsonpath={.items[*].spec.containers[*].name}) ; do echo USDl ; done | egrep "csi-omap-generator|volume-replication"
csi-omap-generator volume-replication csi-omap-generator volume-replication
apiVersion: multicluster.odf.openshift.io/v1alpha1 kind: MirrorPeer metadata: name: mirrorpeer-<cluster1>-<cluster2> spec: items: - clusterName: <cluster1> storageClusterRef: name: ocs-storagecluster namespace: openshift-storage - clusterName: <cluster2> storageClusterRef: name: ocs-storagecluster namespace: openshift-storage
oc patch storagecluster USD(oc get storagecluster -n openshift-storage -o=jsonpath='{.items[0].metadata.name}') -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/mirroring", "value": {"enabled": true} }]'
storagecluster.ocs.openshift.io/ocs-storagecluster patched
oc get cephblockpool -n openshift-storage -o=jsonpath='{.items[?(@.metadata.ownerReferences[*].kind=="StorageCluster")].spec.mirroring.enabled}{"\n"}'
true
oc get pods -o name -l app=rook-ceph-rbd-mirror -n openshift-storage
pod/rook-ceph-rbd-mirror-a-6486c7d875-56v2v
oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'
{"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}