Chapter 12. Restoring the monitor pods in OpenShift Data Foundation Restore the monitor pods if all three of them go down, and when OpenShift Data Foundation is not able to recover the monitor pods automatically. Note This is a disaster recovery procedure and must be performed under the guidance of the Red Hat support team. Contact Red Hat support team on, Red Hat support . Procedure Scale down the rook-ceph-operator and ocs operator deployments. Create a backup of all deployments in openshift-storage namespace. Patch the Object Storage Device (OSD) deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep . Copy tar to the OSDs. Note While copying the tar binary to the OSD, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues. Retrieve the monstore cluster map from all the OSDs. Create the recover_mon.sh script. Run the recover_mon.sh script. Patch the MON deployments, and run it with the command parameter as sleep . Edit the MON deployments. Patch the MON deployments to increase the initialDelaySeconds . Copy tar to the MON pods. Note While copying the tar binary to the MON, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues. Copy the previously retrieved monstore to the mon-a pod. Navigate into the MON pod and change the ownership of the retrieved monstore . Copy the keyring template file before rebuilding the mon db . Populate the keyring of all other Ceph daemons (OSD, MGR, MDS and RGW) from their respective secrets. When getting the daemons keyring, use the following command: Get the OSDs keys with the following script: Copy the mon keyring locally, then edit it by adding all daemon keys captured in the earlier step and copy it back to one of the MON pods (mon-a): As an example, the keyring file should look like the following: Note If the caps entries are not present in the OSDs keys output, make sure to add caps to all the OSDs output as mentioned in the keyring file example. Navigate into the mon-a pod, and verify that the monstore has a monmap . Navigate into the mon-a pod. Verify that the monstore has a monmap . Optional: If the monmap is missing then create a new monmap . <mon-a-id> Is the ID of the mon-a pod. <mon-a-ip> Is the IP address of the mon-a pod. <mon-b-id> Is the ID of the mon-b pod. <mon-b-ip> Is the IP address of the mon-b pod. <mon-c-id> Is the ID of the mon-c pod. <mon-c-ip> Is the IP address of the mon-c pod. <fsid> Is the file system ID. Verify the monmap . Import the monmap . Important Use the previously created keyring file. Create a backup of the old store.db file. Copy the rebuild store.db file to the monstore directory. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods. <id> Is the ID of the MON pod Navigate into the rest of the MON pods and change the ownership of the copied monstore . <id> Is the ID of the MON pod Revert the patched changes. For MON deployments: <mon-deployment.yaml> Is the MON deployment yaml file For OSD deployments: <osd-deployment.yaml> Is the OSD deployment yaml file For MGR deployments: <mgr-deployment.yaml> Is the MGR deployment yaml file Important Ensure that the MON, MGR and OSD pods are up and running. Scale up the rook-ceph-operator and ocs-operator deployments. Verification steps Check the Ceph status to confirm that CephFS is running. Example output: Check the Multicloud Object Gateway (MCG) status. It should be active, and the backingstore and bucketclass should be in Ready state. Important If the MCG is not in the active state, and the backingstore and bucketclass not in the Ready state, you need to restart all the MCG related pods. For more information, see Section 12.1, "Restoring the Multicloud Object Gateway" . 12.1. Restoring the Multicloud Object Gateway If the Multicloud Object Gateway (MCG) is not in the active state, and the backingstore and bucketclass is not in the Ready state, you need to restart all the MCG related pods, and check the MCG status to confirm that the MCG is back up and running. Procedure Restart all the pods related to the MCG. <noobaa-operator> Is the name of the MCG operator <noobaa-core> Is the name of the MCG core pod <noobaa-endpoint> Is the name of the MCG endpoint <noobaa-db> Is the name of the MCG db pod If the RADOS Object Gateway (RGW) is configured, restart the pod. <rgw-pod> Is the name of the RGW pod Note In OpenShift Container Platform 4.11, after the recovery, RBD PVC fails to get mounted on the application pods. Hence, you need to restart the node that is hosting the application pods. To get the node name that is hosting the application pod, run the following command:

Commands:
oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage
oc scale deployment ocs-operator --replicas=0 -n openshift-storage
mkdir backup
cd backup
oc project openshift-storage
for d in USD(oc get deployment|awk -F' ' '{print USD1}'|grep -v NAME); do echo USDd;oc get deployment USDd -o yaml > oc_get_deployment.USD{d}.yaml; done
for i in USD(oc get deployment -l app=rook-ceph-osd -oname);do oc patch USD{i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch USD{i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done
for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i USD{i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i USD{i} -- bash -c 'chmod +x /usr/bin/tar' ;done
#!/bin/bash ms=/tmp/monstore rm -rf USDms mkdir USDms for osd_pod in USD(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do echo "Starting with pod: USDosd_pod" podname=USD(echo USDosd_pod|sed 's/pod\///g') oc exec USDosd_pod -- rm -rf USDms oc exec USDosd_pod -- mkdir USDms oc cp USDms USDpodname:USDms rm -rf USDms mkdir USDms echo "pod in loop: USDosd_pod ; done deleting local dirs" oc exec USDosd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-USD(oc get USDosd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path USDms echo "Done with COT on pod: USDosd_pod" oc cp USDpodname:USDms USDms echo "Finished pulling COT data from pod: USDosd_pod" done
chmod +x recover_mon.sh
./recover_mon.sh
for i in USD(oc get deployment -l app=rook-ceph-mon -oname);do oc patch USD{i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done
for i in a b c ; do oc get deployment rook-ceph-mon-USD{i} -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 10000/g" | oc replace -f - ; done
for i in `oc get pods -l app=rook-ceph-mon -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i USD{i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i USD{i} -- bash -c 'chmod +x /usr/bin/tar' ;done
oc cp /tmp/monstore/ USD(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/
oc rsh USD(oc get po -l app=rook-ceph-mon,mon=a -oname)
chown -R ceph:ceph /tmp/monstore
oc rsh USD(oc get po -l app=rook-ceph-mon,mon=a -oname)
cp /etc/ceph/keyring-store/keyring /tmp/keyring
cat /tmp/keyring [mon.] key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA== caps mon = "allow *" [client.admin] key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson | jq .data.keyring | xargs echo | base64 -d [mds.ocs-storagecluster-cephfilesystem-a] key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA== caps mon = "allow profile mds" caps osd = "allow *" caps mds = "allow"
for i in `oc get secret | grep keyring| awk '{print USD1}'` ; do oc get secret USD{i} -ojson | jq .data.keyring | xargs echo | base64 -d ; done
for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do oc exec -i USD{i} -- bash -c 'cat /var/lib/ceph/osd/ceph-*/keyring ' ;done
cp USD(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/etc/ceph/keyring-store/..data/keyring /tmp/keyring-mon-a
vi /tmp/keyring-mon-a
[mon.] key = AQCbQLRn0j9mKhAAJKWmMZ483QIpMwzx/yGSLw== caps mon = "allow *" [mds.ocs-storagecluster-cephfilesystem-a] key = AQBFQbRnYuB9LxAA8i1fCSAKQQsPuywZ0Jlc5Q== caps mon = "allow profile mds" caps osd = "allow *" caps mds = "allow" [mds.ocs-storagecluster-cephfilesystem-b] key = AQBHQbRnwHAOEBAAv+rBpYP5W8BmC7gLfLyk1w== caps mon = "allow profile mds" caps osd = "allow *" caps mds = "allow" [osd.0] key = AQAvQbRnjF0eEhAA3H0l9zvKGZZM9Up6fJajhQ== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.1] key = AQA0QbRnq4cSGxAA7JpuK1+sq8gALNmMYFUMzw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.2] key = AQA3QbRn6JvcOBAAFKruZQhlQJKUOi9oxcN6fw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [client.admin] key = AQCbQLRnSzOuLBAAK1cSgr2eIyrZV8mV28UfvQ== caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" caps mgr = "allow *" [client.rgw.ocs.storagecluster.cephobjectstore.a] key = AQBTQbRny7NJLRAAPeTvK9kVg71/glbYLANGyw== caps mon = "allow rw" caps osd = "allow rwx" [mgr.a] key = AQD9QLRn8+xzDxAARqWQatoT9ruK76EpDS6iCw== caps mon = "allow profile mgr" caps mds = "allow *" caps osd = "allow *" [mgr.b] key = AQD9QLRnltZOIhAAexshUqdOr3G79HWYXUDGFg== caps mon = "allow profile mgr" caps mds = "allow *" caps osd = "allow *" [client.crash] key = AQD7QLRn6DDzCBAAEzhXRzGQUBUNTzC3nHntFQ== caps mon = "allow profile crash" caps mgr = "allow rw" [client.ceph-exporter] key = AQD7QLRntHzkGxAApQTkMVzcTiZn7jZbwK99SQ== caps mon = "allow profile ceph-exporter" caps mgr = "allow r" caps osd = "allow r" caps mds = "allow r"
cp /tmp/keyring-mon-a USD(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/tmp/keyring
oc rsh USD(oc get po -l app=rook-ceph-mon,mon=a -oname)
ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap
monmaptool /tmp/monmap --print
monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>
monmaptool /root/monmap --print
ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap
chown -R ceph:ceph /tmp/monstore
mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted
mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted
mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted
mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db
chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db
oc cp USD(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db
oc cp /tmp/store.db USD(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph- <id>
oc rsh USD(oc get po -l app=rook-ceph-mon,mon= <id> -oname)
chown -R ceph:ceph /var/lib/ceph/mon/ceph- <id> /store.db
oc replace --force -f <mon-deployment.yaml>
oc replace --force -f <osd-deployment.yaml>
oc replace --force -f <mgr-deployment.yaml>
oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1
oc -n openshift-storage scale deployment ocs-operator --replicas=1
ceph -s
cluster: id: f111402f-84d1-4e06-9fdb-c27607676e55 health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds 3 daemons have recently crashed services: mon: 3 daemons, quorum b,c,a (age 15m) mgr: a(active, since 14m) mds: ocs-storagecluster-cephfilesystem:0 osd: 3 osds: 3 up (since 15m), 3 in (since 2h) data: pools: 3 pools, 96 pgs objects: 500 objects, 1.1 GiB usage: 5.5 GiB used, 295 GiB / 300 GiB avail pgs: 96 active+clean
noobaa status -n openshift-storage
oc delete pods <noobaa-operator> -n openshift-storage
oc delete pods <noobaa-core> -n openshift-storage
oc delete pods <noobaa-endpoint> -n openshift-storage
oc delete pods <noobaa-db> -n openshift-storage
oc delete pods <rgw-pod> -n openshift-storage
oc get pods <application-pod> -n <namespace> -o yaml | grep nodeName nodeName: node_name