Chapter 5. OpenShift Data Foundation deployed using local storage devices 5.1. Replacing operational or failed storage devices on clusters backed by local storage devices You can replace an object storage device (OSD) in OpenShift Data Foundation deployed using local storage devices on the following infrastructures: Bare metal VMware Red Hat Virtualization Note One or more underlying storage devices may need to be replaced. Prerequisites Red Hat recommends that replacement devices are configured with similar infrastructure and resources to the device being replaced. If you upgraded to OpenShift Data Foundation version 4.9 from a version, and have not already created the LocalVolumeDiscovery and LocalVolumeSet objects, follow the procedure described in Post-update configuration changes for clusters backed by local storage . Ensure that the data is resilient. In the OpenShift Web Console, click Storage OpenShift Data Foundation . Click the Storage Systems tab, and then click ocs-storagecluster-storagesystem . In the Status card of Block and File dashboard, under the Overview tab, verify that Data Resiliency has a green tick mark. Procedure Remove the underlying storage device from relevant worker node. Verify that relevant OSD Pod has moved to CrashLoopBackOff state. Identify the OSD that needs to be replaced and the OpenShift Container Platform node that has the OSD scheduled on it. Example output: In this example, rook-ceph-osd-0-6d77d6c7c6-m8xj6 needs to be replaced and compute-2 is the OpenShift Container platform node on which the OSD is scheduled. Scale down the OSD deployment for the OSD to be replaced. where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0 . Example output: Verify that the rook-ceph-osd pod is terminated. Example output: Important If the rook-ceph-osd pod is in terminating state for more than a few minutes, use the force option to delete the pod. Example output: Remove the old OSD from the cluster so that you can add a new OSD. Delete any old ocs-osd-removal jobs. Example output: Navigate to the openshift-storage project. Remove the old OSD from the cluster. <failed_osd_id> Is the integer in the pod name immediately after the rook-ceph-osd prefix. You can add comma separated OSD IDs in the command to remove more than one OSD, for example, FAILED_OSD_IDS=0,1,2 . The FORCE_OSD_REMOVAL value must be changed to true in clusters that only have three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed. Warning This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal-job pod. A status of Completed confirms that the OSD removal job succeeded. Ensure that the OSD removal is completed. Example output: Important If the ocs-osd-removal-job fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example: If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSD devices that are removed from the respective OpenShift Data Foundation nodes. Get the Persistent Volume Claim (PVC) name(s) of the replaced OSD(s) from the logs of ocs-osd-removal-job pod. Example output: For each of the previously identified nodes, do the following: Create a debug pod and chroot to the host on the storage node. <node name> Is the name of the node. Find the relevant device name based on the PVC names identified in the step. <pvc name> Is the name of the PVC. Example output: Remove the mapped device. Important If the above command gets stuck due to insufficient privileges, run the following commands: Press CTRL+Z to exit the above command. Find the PID of the process which was stuck. Terminate the process using the kill command. <PID> Is the process ID. Verify that the device name is removed. Find the persistent volume (PV) that need to be deleted. Example output: Delete the PV. Physically add a new device to the node. Track the provisioning of PVs for the devices that match the deviceInclusionSpec . It can take a few minutes to provision the PVs. Example output: Once the PV is provisioned, a new OSD pod is automatically created for the PV. Delete the ocs-osd-removal job(s). Example output: Note When using an external key management system (KMS) with data encryption, the old OSD encryption key can be removed from the Vault server as it is now an orphan key. Verification steps Verify that there is a new OSD running. Example output: Important If the new OSD does not show as Running after a few minutes, restart the rook-ceph-operator pod to force a reconciliation. Example output: Verify that a new PVC is created. Example output: Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted. Identify the node(s) where the new OSD pod(s) are running. <OSD pod name> Is the name of the OSD pod. For example: For each of the nodes identified in the step, do the following: Create a debug pod and open a chroot environment for the selected host(s). <node name> Is the name of the node. Check for the crypt keyword beside the ocs-deviceset name(s). Log in to OpenShift Web Console and check the OSD status on the storage dashboard. Note A full data recovery may take longer depending on the volume of data being recovered. 5.2. Replacing operational or failed storage devices on IBM Power You can replace an object storage device (OSD) in OpenShift Data Foundation deployed using local storage devices on IBM Power. Note One or more underlying storage devices may need to be replaced. Prerequisites Red Hat recommends that replacement devices are configured with similar infrastructure and resources to the deice being replaced. If you upgraded to OpenShift Data Foundation version 4.9 from a version and have not already created the LocalVolumeDiscovery object, follow the procedure described in Post-update configuration changes for clusters backed by local storage . Ensure that the data is resilient. In the OpenShift Web Console, click Storage OpenShift Data Foundation . Click the Storage Systems tab, and then click ocs-storagecluster-storagesystem . In the Status card of Block and File dashboard, under the Overview tab, verify that Data Resiliency has a green tick mark. Procedure Identify the OSD that needs to be replaced and the OpenShift Container Platform node that has the OSD scheduled on it. Example output: In this example, rook-ceph-osd-0-86bf8cdc8-4nb5t needs to be replaced and worker-0 is the RHOCP node on which the OSD is scheduled. Note If the OSD to be replaced is healthy, the status of the pod will be Running . Scale down the OSD deployment for the OSD to be replaced. where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0 . Example output: Verify that the rook-ceph-osd pod is terminated. Example output: Important If the rook-ceph-osd pod is in terminating state for more than a few minutes, use the force option to delete the pod. Example output: Remove the old OSD from the cluster so that you can add a new OSD. Identify the DeviceSet associated with the OSD to be replaced. Example output: In this example, the Persistent Volume Claim (PVC) name is ocs-deviceset-localblock-0-data-0-64xjl . Identify the Persistent Volume (PV) associated with the PVC. where, x , y , and pvc-suffix are the values in the DeviceSet identified in an earlier step. Example output: In this example, the associated PV is local-pv-8137c873 . Identify the name of the device to be replaced. where, pv-suffix is the value in the PV name identified in an earlier step. Example output: In this example, the device name is vdc . Identify the prepare-pod associated with the OSD to be replaced. where, x , y , and pvc-suffix are the values in the DeviceSet identified in an earlier step. Example output: In this example, the prepare-pod name is rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data-0-64knzkc . Delete any old ocs-osd-removal jobs. Example output: Change to the openshift-storage project. Remove the old OSD from the cluster. <failed_osd_id> Is the integer in the pod name immediately after the rook-ceph-osd prefix. You can add comma separated OSD IDs in the command to remove more than one OSD, for example, FAILED_OSD_IDS=0,1,2 . The FORCE_OSD_REMOVAL value must be changed to true in clusters that only have three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed. Warning This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal-job pod. A status of Completed confirms that the OSD removal job succeeded. Ensure that the OSD removal is completed. Example output: Important If the ocs-osd-removal-job fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example: If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSD devices that are removed from the respective OpenShift Data Foundation nodes. Get the PVC name(s) of the replaced OSD(s) from the logs of ocs-osd-removal-job pod. Example output: For each of the previously identified nodes, do the following: Create a debug pod and chroot to the host on the storage node. <node name> Is the name of the node. Find the relevant device name based on the PVC names identified in the step. <pvc name> Is the name of the PVC. Example output: Remove the mapped device. Important If the above command gets stuck due to insufficient privileges, run the following commands: Press CTRL+Z to exit the above command. Find the PID of the process which was stuck. Terminate the process using the kill command. <PID> Is the process ID. Verify that the device name is removed. Find the PV that need to be deleted. Example output: Delete the PV. <pv-name> Is the name of the PV. Replace the old device and use the new device to create a new OpenShift Container Platform PV. Log in to the OpenShift Container Platform node with the device to be replaced. In this example, the OpenShift Container Platform node is worker-0 . Example output: Record the /dev/disk that is to be replaced using the device name, vdc , identified earlier. Example output: Find the name of the LocalVolume CR, and remove or comment out the device /dev/disk that is to be replaced. Example output: Example output: Make sure to save the changes after editing the CR. Log in to the OpenShift Container Platform node with the device to be replaced and remove the old symlink . Example output: Identify the old symlink for the device name to be replaced. In this example, the device name is vdc . Example output: Remove the symlink . Verify that the symlink is removed. Example output: Replace the old device with the new device. Log back into the correct OpenShift Cotainer Platform node and identify the device name for the new drive. The device name must change unless you are resetting the same device. Example output: In this example, the new device name is vdd . After the new /dev/disk is available, you can add a new disk entry to the LocalVolume CR. Edit the LocalVolume CR and add the new /dev/disk . In this example, the new device is /dev/vdd . Example output: Make sure to save the changes after editing the CR. Verify that there is a new PV in Available state and of the correct size. Example output: Create a new OSD for the new device. Deploy the new OSD. You need to restart the rook-ceph-operator to force operator reconciliation. Identify the name of the rook-ceph-operator . Example output: Delete the rook-ceph-operator . Example output: In this example, the rook-ceph-operator pod name is rook-ceph-operator-85f6494db4-sg62v . Verify that the rook-ceph-operator pod is restarted. Example output: Creation of the new OSD may take several minutes after the operator restarts. Delete the ocs-osd-removal job(s). Example output: Note When using an external key management system (KMS) with data encryption, the old OSD encryption key can be removed from the Vault server as it is now an orphan key. Verfication steps Verify that there is a new OSD running. Example output: Verify that a new PVC is created. Example output: Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted. Identify the node(s) where the new OSD pod(s) are running. <OSD pod name> Is the name of the OSD pod. For example: For each of the previously identified nodes, do the following: Create a debug pod and open a chroot environment for the selected host(s). <node name> Is the name of the node. Check for the crypt keyword beside the ocs-deviceset name(s). Log in to OpenShift Web Console and check the status card in the Openshift Data Foundation dashboard under Storage section. Note A full data recovery may take longer depending on the volume of data being recovered. 5.3. Replacing operational or failed storage devices on IBM Z or LinuxONE infrastructure You can replace operational or failed storage devices on IBM Z or LinuxONE infrastructure with new Small Computer System Interface (SCSI) disks. IBM Z or LinuxONE supports SCSI FCP disk logical units (SCSI disks) as persistent storage devices from external disk storage. You can identify a SCSI disk using its FCP Device number, two target worldwide port names (WWPN1 and WWPN2), and the logical unit number (LUN). For more information, see https://www.ibm.com/support/knowledgecenter/SSB27U_6.4.0/com.ibm.zvm.v640.hcpa5/scsiover.html Prerequisites Ensure that the data is resilient. In the OpenShift Web Console, click Storage OpenShift Data Foundation . Click the Storage Systems tab, and then click ocs-storagecluster-storagesystem . In the Status card of Block and File dashboard, under the Overview tab, verify that Data Resiliency has a green tick mark. Procedure List all the disks. Example output: A SCSI disk is represented as a zfcp-lun with the structure <device-id>:<wwpn>:<lun-id> in the ID section. The first disk is used for the operating system. If one storage device fails, you can replace it with a new disk. Remove the disk. Run the following command on the disk, replacing scsi-id with the SCSI disk identifier of the disk to be replaced: For example, the following command removes one disk with the device ID 0.0.8204 , the WWPN 0x500507630a0b50a4 , and the LUN 0x4002403000000000 : Append a new SCSI disk. Note The device ID for the new disk must be the same as the disk to be replaced. The new disk is identified with its WWPN and LUN ID. List all the FCP devices to verify the new disk is configured. Example output:

Commands:
oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide
rook-ceph-osd-0-6d77d6c7c6-m8xj6 0/1 CrashLoopBackOff 0 24h 10.129.0.16 compute-2 <none> <none> rook-ceph-osd-1-85d99fb95f-2svc7 1/1 Running 0 24h 10.128.2.24 compute-0 <none> <none> rook-ceph-osd-2-6c66cdb977-jp542 1/1 Running 0 24h 10.130.0.18 compute-1 <none> <none>
osd_id_to_remove=0
oc scale -n openshift-storage deployment rook-ceph-osd-USD{osd_id_to_remove} --replicas=0
deployment.extensions/rook-ceph-osd-0 scaled
oc get -n openshift-storage pods -l ceph-osd-id=USD{osd_id_to_remove}
No resources found in openshift-storage namespace.
oc delete -n openshift-storage pod rook-ceph-osd-0-6d77d6c7c6-m8xj6 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "rook-ceph-osd-0-6d77d6c7c6-m8xj6" force deleted
oc delete -n openshift-storage job ocs-osd-removal-job
job.batch "ocs-osd-removal-job" deleted
oc project openshift-storage
oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS= <failed_osd_id> FORCE_OSD_REMOVAL=false | oc create -n openshift-storage -f -
oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'
2022-05-10 06:50:04.501511 I | cephosd: completed removal of OSD 0
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 |egrep -i 'pvc|deviceset'
2021-05-12 14:31:34.666000 I | cephosd: removing the OSD PVC "ocs-deviceset-xxxx-xxx-xxx-xxx"
oc debug node/ <node name>
chroot /host
dmsetup ls| grep <pvc name>
ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt (253:0)
cryptsetup luksClose --debug --verbose ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt
ps -ef | grep crypt
kill -9 <PID>
dmsetup ls
oc get pv -L kubernetes.io/hostname | grep localblock | grep Released
local-pv-d6bf175b 1490Gi RWO Delete Released openshift-storage/ocs-deviceset-0-data-0-6c5pw localblock 2d22h compute-1
oc delete pv <pv_name>
oc -n openshift-local-storage describe localvolumeset localblock
[...] Status: Conditions: Last Transition Time: 2020-11-17T05:03:32Z Message: DiskMaker: Available, LocalProvisioner: Available Status: True Type: DaemonSetsAvailable Last Transition Time: 2020-11-17T05:03:34Z Message: Operator reconciled successfully. Status: True Type: Available Observed Generation: 1 Total Provisioned Device Count: 4 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Discovered 2m30s (x4 localvolumeset- node.example.com - NewDevice over 2m30s) symlink-controller found possible matching disk, waiting 1m to claim Normal FoundMatch 89s (x4 localvolumeset- node.example.com - ingDisk over 89s) symlink-controller symlinking matching disk
oc delete -n openshift-storage job ocs-osd-removal-job
job.batch "ocs-osd-removal-job" deleted
oc get -n openshift-storage pods -l app=rook-ceph-osd
rook-ceph-osd-0-5f7f4747d4-snshw 1/1 Running 0 4m47s rook-ceph-osd-1-85d99fb95f-2svc7 1/1 Running 0 1d20h rook-ceph-osd-2-6c66cdb977-jp542 1/1 Running 0 1d20h
oc delete pod -n openshift-storage -l app=rook-ceph-operator
pod "rook-ceph-operator-6f74fb5bff-2d982" deleted
oc get -n openshift-storage pvc | grep localblock
ocs-deviceset-0-0-c2mqb Bound local-pv-b481410 1490Gi RWO localblock 5m ocs-deviceset-1-0-959rp Bound local-pv-414755e0 1490Gi RWO localblock 1d20h ocs-deviceset-2-0-79j94 Bound local-pv-3e8964d3 1490Gi RWO localblock 1d20h
oc get -o=custom-columns=NODE:.spec.nodeName pod/ <OSD pod name>
get -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm
oc debug node/ <node name>
chroot /host
lsblk
oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide
rook-ceph-osd-0-86bf8cdc8-4nb5t 0/1 crashLoopBackOff 0 24h 10.129.2.26 worker-0 <none> <none> rook-ceph-osd-1-7c99657cfb-jdzvz 1/1 Running 0 24h 10.128.2.46 worker-1 <none> <none> rook-ceph-osd-2-5f9f6dfb5b-2mnw9 1/1 Running 0 24h 10.131.0.33 worker-2 <none> <none>
osd_id_to_remove=0
oc scale -n openshift-storage deployment rook-ceph-osd-USD{osd_id_to_remove} --replicas=0
deployment.extensions/rook-ceph-osd-0 scaled
oc get -n openshift-storage pods -l ceph-osd-id=USD{osd_id_to_remove}
No resources found in openshift-storage namespace.
oc delete -n openshift-storage pod rook-ceph-osd-0-86bf8cdc8-4nb5t --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "rook-ceph-osd-0-86bf8cdc8-4nb5t" force deleted
oc get -n openshift-storage -o yaml deployment rook-ceph-osd-USD{osd_id_to_remove} | grep ceph.rook.io/pvc
ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl
oc get -n openshift-storage pvc ocs-deviceset- <x> - <y> - <pvc-suffix>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ocs-deviceset-localblock-0-data-0-64xjl Bound local-pv-8137c873 256Gi RWO localblock 24h
oc get pv local-pv- <pv-suffix> -o yaml | grep path
path: /mnt/local-storage/localblock/vdc
oc describe -n openshift-storage pvc ocs-deviceset- <x> - <y> - <pvc-suffix> | grep Used
Used By: rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data-0-64knzkc
oc delete -n openshift-storage job ocs-osd-removal-job
job.batch "ocs-osd-removal-job" deleted
oc project openshift-storage
oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS= <failed_osd_id> FORCE_OSD_REMOVAL=false | oc create -n openshift-storage -f -
oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'
2022-05-10 06:50:04.501511 I | cephosd: completed removal of OSD 0
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 |egrep -i 'pvc|deviceset'
2021-05-12 14:31:34.666000 I | cephosd: removing the OSD PVC "ocs-deviceset-xxxx-xxx-xxx-xxx"
oc debug node/ <node name>
chroot /host
dmsetup ls| grep <pvc name>
ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt (253:0)
cryptsetup luksClose --debug --verbose ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt
ps -ef | grep crypt
kill -9 <PID>
dmsetup ls
oc get pv -L kubernetes.io/hostname | grep localblock | grep Released
local-pv-d6bf175b 1490Gi RWO Delete Released openshift-storage/ocs-deviceset-0-data-0-6c5pw localblock 2d22h compute-1
oc delete pv <pv-name>
oc debug node/worker-0
Starting pod/worker-0-debug To use host binaries, run `chroot /host` Pod IP: 192.168.88.21 If you don't see a command prompt, try pressing enter. chroot /host
ls -alh /mnt/local-storage/localblock
total 0 drwxr-xr-x. 2 root root 17 Nov 18 15:23 . drwxr-xr-x. 3 root root 24 Nov 18 15:23 .. lrwxrwxrwx. 1 root root 8 Nov 18 15:23 vdc -> /dev/vdc
oc get -n openshift-local-storage localvolume
NAME AGE localblock 25h
oc edit -n openshift-local-storage localvolume localblock
[...] storageClassDevices: - devicePaths: # - /dev/vdc storageClassName: localblock volumeMode: Block [...]
oc debug node/worker-0
Starting pod/worker-0-debug To use host binaries, run `chroot /host` Pod IP: 192.168.88.21 If you don't see a command prompt, try pressing enter. chroot /host
ls -alh /mnt/local-storage/localblock
total 0 drwxr-xr-x. 2 root root 17 Nov 18 15:23 . drwxr-xr-x. 3 root root 24 Nov 18 15:23 .. lrwxrwxrwx. 1 root root 8 Nov 18 15:23 vdc -> /dev/vdc
rm /mnt/local-storage/localblock/vdc
ls -alh /mnt/local-storage/localblock
total 0 drwxr-xr-x. 2 root root 6 Nov 18 17:11 . drwxr-xr-x. 3 root root 24 Nov 18 15:23 ..
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 40G 0 disk |-vda1 252:1 0 4M 0 part |-vda2 252:2 0 384M 0 part /boot `-vda4 252:4 0 39.6G 0 part `-coreos-luks-root-nocrypt 253:0 0 39.6G 0 dm /sysroot vdb 252:16 0 512B 1 disk vdd 252:32 0 256G 0 disk
oc edit -n openshift-local-storage localvolume localblock
[...] storageClassDevices: - devicePaths: # - /dev/vdc - /dev/vdd storageClassName: localblock volumeMode: Block [...]
oc get pv | grep 256Gi
local-pv-1e31f771 256Gi RWO Delete Bound openshift-storage/ocs-deviceset-localblock-2-data-0-6xhkf localblock 24h local-pv-ec7f2b80 256Gi RWO Delete Bound openshift-storage/ocs-deviceset-localblock-1-data-0-hr2fx localblock 24h local-pv-8137c873 256Gi RWO Delete Available localblock 32m
oc get -n openshift-storage pod -l app=rook-ceph-operator
NAME READY STATUS RESTARTS AGE rook-ceph-operator-85f6494db4-sg62v 1/1 Running 0 1d20h
oc delete -n openshift-storage pod rook-ceph-operator-85f6494db4-sg62v
pod "rook-ceph-operator-85f6494db4-sg62v" deleted
oc get -n openshift-storage pod -l app=rook-ceph-operator
NAME READY STATUS RESTARTS AGE rook-ceph-operator-85f6494db4-wx9xx 1/1 Running 0 50s
oc delete -n openshift-storage job ocs-osd-removal-job
job.batch "ocs-osd-removal-job" deleted
oc get -n openshift-storage pods -l app=rook-ceph-osd
rook-ceph-osd-0-76d8fb97f9-mn8qz 1/1 Running 0 23m rook-ceph-osd-1-7c99657cfb-jdzvz 1/1 Running 1 25h rook-ceph-osd-2-5f9f6dfb5b-2mnw9 1/1 Running 0 25h
oc get -n openshift-storage pvc | grep localblock
ocs-deviceset-localblock-0-data-0-q4q6b Bound local-pv-8137c873 256Gi RWO localblock 10m ocs-deviceset-localblock-1-data-0-hr2fx Bound local-pv-ec7f2b80 256Gi RWO localblock 1d20h ocs-deviceset-localblock-2-data-0-6xhkf Bound local-pv-1e31f771 256Gi RWO localblock 1d20h
oc get -o=custom-columns=NODE:.spec.nodeName pod/ <OSD pod name>
get -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm
oc debug node/ <node name>
chroot /host
lsblk
lszdev
TYPE ID zfcp-host 0.0.8204 yes yes zfcp-lun 0.0.8204:0x102107630b1b5060:0x4001402900000000 yes no sda sg0 zfcp-lun 0.0.8204:0x500407630c0b50a4:0x3002b03000000000 yes yes sdb sg1 qeth 0.0.bdd0:0.0.bdd1:0.0.bdd2 yes no encbdd0 generic-ccw 0.0.0009 yes no
chzdev -d scsi-id
chzdev -d 0.0.8204:0x500407630c0b50a4:0x3002b03000000000
chzdev -e 0.0.8204:0x500507630b1b50a4:0x4001302a00000000
lszdev zfcp-lun
TYPE ID ON PERS NAMES zfcp-lun 0.0.8204:0x102107630b1b5060:0x4001402900000000 yes no sda sg0 zfcp-lun 0.0.8204:0x500507630b1b50a4:0x4001302a00000000 yes yes sdb sg1