Chapter 9. Troubleshooting Ceph objects As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level or low-level object operations. The ceph-objectstore-tool utility can help you troubleshoot problems related to objects within a particular OSD or placement group. Important Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility. 9.1. Prerequisites Verify there are no network-related issues. 9.2. Troubleshooting high-level object operations As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level object operations. The ceph-objectstore-tool utility supports the following high-level object operations: List objects List lost objects Fix lost objects Important Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility. 9.2.1. Prerequisites Root-level access to the Ceph OSD nodes. 9.2.2. Listing objects The OSD can contain zero to many placement groups, and zero to many objects within a placement group (PG). The ceph-objectstore-tool utility allows you to list objects stored within an OSD. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Log in to the OSD container: Syntax Example Identify all the objects within an OSD, regardless of their placement group: Syntax Example Identify all the objects within a placement group: Syntax Example Identify the PG an object belongs to: Syntax Example 9.2.3. Fixing lost objects You can use the ceph-objectstore-tool utility to list and fix lost and unfound objects stored within a Ceph OSD. This procedure applies only to legacy objects. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Log in to the OSD container: Syntax Example To list all the lost legacy objects: Syntax Example Use the ceph-objectstore-tool utility to fix lost and unfound objects. Select the appropriate circumstance: To fix all lost objects: Syntax Example To fix all the lost objects within a placement group: Syntax Example To fix a lost object by its identifier: Syntax Example 9.3. Troubleshooting low-level object operations As a storage administrator, you can use the ceph-objectstore-tool utility to perform low-level object operations. The ceph-objectstore-tool utility supports the following low-level object operations: Manipulate the object's content Remove an object List the object map (OMAP) Manipulate the OMAP header Manipulate the OMAP key List the object's attributes Manipulate the object's attribute key Important Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility. 9.3.1. Prerequisites Root-level access to the Ceph OSD nodes. 9.3.2. Manipulating the object's content With the ceph-objectstore-tool utility, you can get or set bytes on an object. Important Setting the bytes on an object can cause unrecoverable data loss. To prevent data loss, make a backup copy of the object. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Find the object by listing the objects of the OSD or placement group (PG). Log in to the OSD container: Syntax Example Before setting the bytes on an object, make a backup and a working copy of the object: Syntax Example Edit the working copy object file and modify the object contents accordingly. Set the bytes of the object: Syntax Example 9.3.3. Removing an object Use the ceph-objectstore-tool utility to remove an object. By removing an object, its contents and references are removed from the placement group (PG). Important You cannot recreate an object once it is removed. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Log in to the OSD container: Syntax Example Remove an object: Syntax Example 9.3.4. Listing the object map Use the ceph-objectstore-tool utility to list the contents of the object map (OMAP). The output provides you a list of keys. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Example Log in to the OSD container: Syntax Example List the object map: Syntax Example 9.3.5. Manipulating the object map header The ceph-objectstore-tool utility outputs the object map (OMAP) header with the values associated with the object's keys. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Log in to the OSD container: Syntax Example Get the object map header: Syntax Example Set the object map header: Syntax Example 9.3.6. Manipulating the object map key Use the ceph-objectstore-tool utility to change the object map (OMAP) key. You need to provide the data path, the placement group identifier (PG ID), the object, and the key in the OMAP. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Log in to the OSD container: Syntax Example Get the object map key: Syntax Example Set the object map key: Syntax Example Remove the object map key: Syntax Example 9.3.7. Listing the object's attributes Use the ceph-objectstore-tool utility to list an object's attributes. The output provides you with the object's keys and values. Prerequisites Root-level access to the Ceph OSD node. Stopping the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Log in to the OSD container: Syntax Example List the object's attributes: Syntax Example 9.3.8. Manipulating the object attribute key Use the ceph-objectstore-tool utility to change an object's attributes. To manipulate the object's attributes you need the data paths, the placement group identifier (PG ID), the object, and the key in the object's attribute. Prerequisites Root-level access to the Ceph OSD node. Stop the ceph-osd daemon. Procedure Verify the appropriate OSD is down: Syntax Example Log in to the OSD container: Syntax Example Get the object's attributes: Syntax Example Set an object's attributes: Syntax Example Remove an object's attributes: Syntax Example 9.4. Additional Resources For Red Hat Ceph Storage support, see the Red Hat Customer Portal .

Commands:
systemctl status ceph- FSID @osd. OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --op list
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list
ceph-objectstore-tool --data-path PATH_TO_OSD --op list OBJECT_ID
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list default.region
systemctl status ceph- FSID @osd. OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost --dry-run
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost --dry-run
ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op fix-lost
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op fix-lost
ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost OBJECT_ID
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost default.region
systemctl status ceph-osd@ OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT get-bytes > OBJECT_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' get-bytes > zone_info.default.backup ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' get-bytes > zone_info.default.working-copy
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT set-bytes < OBJECT_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' set-bytes < zone_info.default.working-copy
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT remove
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' remove
systemctl status ceph-osd@ OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT list-omap
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' list-omap
systemctl status ceph- FSID @osd. OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT get-omaphdr > OBJECT_MAP_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' get-omaphdr > zone_info.default.omaphdr.txt
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT get-omaphdr < OBJECT_MAP_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' set-omaphdr < zone_info.default.omaphdr.txt
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT get-omap KEY > OBJECT_MAP_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' get-omap "" > zone_info.default.omap.txt
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT set-omap KEY < OBJECT_MAP_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' set-omap "" < zone_info.default.omap.txt
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT rm-omap KEY
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' rm-omap ""
systemctl status ceph- FSID @osd. OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT list-attrs
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' list-attrs
systemctl status ceph- FSID @osd. OSD_ID
systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
cephadm shell --name osd. OSD_ID
cephadm shell --name osd.0
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT get-attr KEY > OBJECT_ATTRS_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' get-attr "oid" > zone_info.default.attr.txt
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT set-attr KEY < OBJECT_ATTRS_FILE_NAME
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' set-attr "oid" < zone_info.default.attr.txt
ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID OBJECT rm-attr KEY
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' rm-attr "oid"