Chapter 4. Troubleshooting Ceph Monitors This chapter contains information on how to fix the most common errors related to the Ceph Monitors. 4.1. Prerequisites Verify the network connection. 4.2. Most common Ceph Monitor errors The following tables list the most common error messages that are returned by the ceph health detail command, or included in the Ceph logs. The tables provide links to corresponding sections that explain the errors and point to specific procedures to fix the problems. 4.2.1. Prerequisites A running Red Hat Ceph Storage cluster. 4.2.2. Ceph Monitor error messages A table of common Ceph Monitor error messages, and a potential fix. Error message See HEALTH_WARN mon.X is down (out of quorum) Ceph Monitor is out of quorum clock skew Clock skew store is getting too big! The Ceph Monitor store is getting too big 4.2.3. Common Ceph Monitor error messages in the Ceph logs A table of common Ceph Monitor error messages found in the Ceph logs, and a link to a potential fix. Error message Log file See clock skew Main cluster log Clock skew clocks not synchronized Main cluster log Clock skew Corruption: error in middle of record Monitor log Ceph Monitor is out of quorum Recovering the Ceph Monitor store Corruption: 1 missing files Monitor log Ceph Monitor is out of quorum Recovering the Ceph Monitor store Caught signal (Bus error) Monitor log Ceph Monitor is out of quorum 4.2.4. Ceph Monitor is out of quorum One or more Ceph Monitors are marked as down but the other Ceph Monitors are still able to form a quorum. In addition, the ceph health detail command returns an error message similar to the following one: What This Means Ceph marks a Ceph Monitor as down due to various reasons. If the ceph-mon daemon is not running, it might have a corrupted store or some other error is preventing the daemon from starting. Also, the /var/ partition might be full. As a consequence, ceph-mon is not able to perform any operations to the store located by default at /var/lib/ceph/mon- SHORT_HOST_NAME /store.db and terminates. If the ceph-mon daemon is running but the Ceph Monitor is out of quorum and marked as down , the cause of the problem depends on the Ceph Monitor state: If the Ceph Monitor is in the probing state longer than expected, it cannot find the other Ceph Monitors. This problem can be caused by networking issues, or the Ceph Monitor can have an outdated Ceph Monitor map ( monmap ) and be trying to reach the other Ceph Monitors on incorrect IP addresses. Alternatively, if the monmap is up-to-date, Ceph Monitor's clock might not be synchronized. If the Ceph Monitor is in the electing state longer than expected, the Ceph Monitor's clock might not be synchronized. If the Ceph Monitor changes its state from synchronizing to electing and back, the cluster state is advancing. This means that it is generating new maps faster than the synchronization process can handle. If the Ceph Monitor marks itself as the leader or a peon , then it believes to be in a quorum, while the remaining cluster is sure that it is not. This problem can be caused by failed clock synchronization. To Troubleshoot This Problem Verify that the ceph-mon daemon is running. If not, start it: Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure. If you are not able to start ceph-mon , follow the steps in The ceph-mon daemon cannot start . If you are able to start the ceph-mon daemon but is is marked as down , follow the steps in The ceph-mon daemon is running, but marked as `down` . The ceph-mon Daemon Cannot Start Check the corresponding Ceph Monitor log, by default located at /var/log/ceph/ceph-mon. HOST_NAME .log . If the log contains error messages similar to the following ones, the Ceph Monitor might have a corrupted store. To fix this problem, replace the Ceph Monitor. See Replacing a failed monitor . If the log contains an error message similar to the following one, the /var/ partition might be full. Delete any unnecessary data from /var/ . Important Do not delete any data from the Monitor directory manually. Instead, use the ceph-monstore-tool to compact it. See Compacting the Ceph Monitor store for details. If you see any other error messages, open a support ticket. See Contacting Red Hat Support for service for details. The ceph-mon Daemon Is Running, but Still Marked as down From the Ceph Monitor host that is out of the quorum, use the mon_status command to check its state: Replace ID with the ID of the Ceph Monitor, for example: If the status is probing , verify the locations of the other Ceph Monitors in the mon_status output. If the addresses are incorrect, the Ceph Monitor has incorrect Ceph Monitor map ( monmap ). To fix this problem, see Injecting a Ceph Monitor map . If the addresses are correct, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. In addition, troubleshoot any networking issues, see Troubleshooting Networking issues for details. If the status is electing , verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. If the status changes from electing to synchronizing , open a support ticket. See Contacting Red Hat Support for service for details. If the Ceph Monitor is the leader or a peon , verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. Open a support ticket if synchronizing the clocks does not solve the problem. See Contacting Red Hat Support for service for details. Additional Resources See Understanding Ceph Monitor status The Starting, Stopping, Restarting the Ceph daemons by instance section in the Administration Guide for Red Hat Ceph Storage 4 The Using the Ceph Administration Socket section in the Administration Guide for Red Hat Ceph Storage 4 4.2.5. Clock skew A Ceph Monitor is out of quorum, and the ceph health detail command output contains error messages similar to these: In addition, Ceph logs contain error messages similar to these: What This Means The clock skew error message indicates that Ceph Monitors' clocks are not synchronized. Clock synchronization is important because Ceph Monitors depend on time precision and behave unpredictably if their clocks are not synchronized. The mon_clock_drift_allowed parameter determines what disparity between the clocks is tolerated. By default, this parameter is set to 0.05 seconds. Important Do not change the default value of mon_clock_drift_allowed without testing. Changing this value might affect the stability of the Ceph Monitors and the Ceph Storage Cluster in general. Possible causes of the clock skew error include network problems or problems with Network Time Protocol (NTP) synchronization if that is configured. In addition, time synchronization does not work properly on Ceph Monitors deployed on virtual machines. To Troubleshoot This Problem Verify that your network works correctly. For details, see Troubleshooting networking issues . In particular, troubleshoot any problems with NTP clients if you use NTP. If you use chrony for NTP, see Basic chrony NTP troubleshooting section for more information. If you use ntpd , see Basic NTP troubleshooting . If you use a remote NTP server, consider deploying your own NTP server on your network. For details, see the Using the Chrony suite to configure NTP chapter in the Configuring basic system settings for Red Hat Enterprise Linux 8. See the Configuring NTP Using ntpd chapter in the System Administrator's Guide for Red Hat Enterprise Linux 7. Note Ceph evaluates time synchronization every five minutes only so there will be a delay between fixing the problem and clearing the clock skew messages. Additional Resources Understanding Ceph Monitor status Ceph Monitor is out of quorum 4.2.6. The Ceph Monitor store is getting too big The ceph health command returns an error message similar to the following one: What This Means Ceph Monitors store is in fact a LevelDB database that stores entries as key-values pairs. The database includes a cluster map and is located by default at /var/lib/ceph/mon/ CLUSTER_NAME - SHORT_HOST_NAME /store.db . Querying a large Monitor store can take time. As a consequence, the Ceph Monitor can be delayed in responding to client queries. In addition, if the /var/ partition is full, the Ceph Monitor cannot perform any write operations to the store and terminates. See Ceph Monitor is out of quorum for details on troubleshooting this issue. To Troubleshoot This Problem Check the size of the database: Specify the name of the cluster and the short host name of the host where the ceph-mon is running. Example Compact the Ceph Monitor store. For details, see Compacting the Ceph Monitor Store . Additional Resources Ceph Monitor is out of quorum 4.2.7. Understanding Ceph Monitor status The mon_status command returns information about a Ceph Monitor, such as: State Rank Elections epoch Monitor map ( monmap ) If Ceph Monitors are able to form a quorum, use mon_status with the ceph command-line utility. If Ceph Monitors are not able to form a quorum, but the ceph-mon daemon is running, use the administration socket to execute mon_status . An example output of mon_status Ceph Monitor States Leader During the electing phase, Ceph Monitors are electing a leader. The leader is the Ceph Monitor with the highest rank, that is the rank with the lowest value. In the example above, the leader is mon.1 . Peon Peons are the Ceph Monitors in the quorum that are not leaders. If the leader fails, the peon with the highest rank becomes a new leader. Probing A Ceph Monitor is in the probing state if it is looking for other Ceph Monitors. For example after you start the Ceph Monitors, they are probing until they find enough Ceph Monitors specified in the Ceph Monitor map ( monmap ) to form a quorum. Electing A Ceph Monitor is in the electing state if it is in the process of electing the leader. Usually, this status changes quickly. Synchronizing A Ceph Monitor is in the synchronizing state if it is synchronizing with the other Ceph Monitors to join the quorum. The smaller the Ceph Monitor store it, the faster the synchronization process. Therefore, if you have a large store, synchronization takes longer time. Additional Resources For details, see the Using the Ceph Administration Socket section in the Administration Guide for Red Hat Ceph Storage 4. 4.2.8. Additional Resources See the Section 4.2.2, "Ceph Monitor error messages" in the Red Hat Ceph Storage Troubleshooting Guide . See the Section 4.2.3, "Common Ceph Monitor error messages in the Ceph logs" in the Red Hat Ceph Storage Troubleshooting Guide . 4.3. Injecting a monmap If a Ceph Monitor has an outdated or corrupted Ceph Monitor map ( monmap ), it cannot join a quorum because it is trying to reach the other Ceph Monitors on incorrect IP addresses. The safest way to fix this problem is to obtain and inject the actual Ceph Monitor map from other Ceph Monitors. Note This action overwrites the existing Ceph Monitor map kept by the Ceph Monitor. This procedure shows how to inject the Ceph Monitor map when the other Ceph Monitors are able to form a quorum, or when at least one Ceph Monitor has a correct Ceph Monitor map. If all Ceph Monitors have corrupted store and therefore also the Ceph Monitor map, see Recovering the Ceph Monitor store . Prerequisites Access to the Ceph Monitor Map. Root-level access to the Ceph Monitor node. Procedure If the remaining Ceph Monitors are able to form a quorum, get the Ceph Monitor map by using the ceph mon getmap command: If the remaining Ceph Monitors are not able to form the quorum and you have at least one Ceph Monitor with a correct Ceph Monitor map, copy it from that Ceph Monitor: Stop the Ceph Monitor which you want to copy the Ceph Monitor map from: For example, to stop the Ceph Monitor running on a host with the host1 short host name: Copy the Ceph Monitor map: Replace ID with the ID of the Ceph Monitor which you want to copy the Ceph Monitor map from: Stop the Ceph Monitor with the corrupted or outdated Ceph Monitor map: For example, to stop a Ceph Monitor running on a host with the host2 short host name: You can inject the Ceph Monitor map as a ceph user in two different ways: Run the command as a ceph user: Syntax Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map: Example Run the command as a root user and then run chown to change the permissions: Run the command as a root user: Syntax Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map: Example Change the file permissions: Example Start the Ceph Monitor: If you copied the Ceph Monitor map from another Ceph Monitor, start that Ceph Monitor, too: Additional Resources Ceph Monitor is out of quorum Recovering the Ceph Monitor store 4.4. Replacing a failed Monitor When a Monitor has a corrupted store, the recommended way to fix this problem is to replace the Monitor by using the Ansible automation application. Prerequisites A running Red Hat Ceph Storage cluster. Able to form a quroum. Root-level access to Ceph Monitor node. Procedure From the Monitor host, remove the Monitor store by default located at /var/lib/ceph/mon/ CLUSTER_NAME - SHORT_HOST_NAME : Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor store of a Monitor running on host1 from a cluster called remote : Remove the Monitor from the Monitor map ( monmap ): Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor running on host1 from a cluster called remote : Troubleshoot and fix any problems related to the underlying file system or hardware of the Monitor host. From the Ansible administration node, redeploy the Monitor by running the ceph-ansible playbook: Additional Resources See the Ceph Monitor is out of quorum for details. The Managing the storage cluster size chapter in the Red Hat Ceph Storage Operations Guide . The Deploying Red Hat Ceph Storage chapter in the Red Hat Ceph Storage 4 Installation Guide . 4.5. Compacting the monitor store When the Monitor store has grown big in size, you can compact it: Dynamically by using the ceph tell command. Upon the start of the ceph-mon daemon. By using the ceph-monstore-tool when the ceph-mon daemon is not running. Use this method when the previously mentioned methods fail to compact the Monitor store or when the Monitor is out of quorum and its log contains the Caught signal (Bus error) error message. Important Monitor store size changes when the cluster is not in the active+clean state or during the rebalancing process. For this reason, compact the Monitor store when rebalancing is completed. Also, ensure that the placement groups are in the active+clean state. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to the Ceph Monitor node. Procedure To compact the Monitor store when the ceph-mon daemon is running: Replace HOST_NAME with the short host name of the host where the ceph-mon is running. Use the hostname -s command when unsure. Add the following parameter to the Ceph configuration under the [mon] section: Restart the ceph-mon daemon: Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure. Ensure that Monitors have formed a quorum: Repeat these steps on other Monitors if needed. Note Before you start, ensure that you have the ceph-test package installed. Verify that the ceph-mon daemon with the large store is not running. Stop the daemon if needed. Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure. Compact the Monitor store as a ceph user in two different ways: Run the command as a ceph user: Syntax Example Run the command as a root user and then run chown to change the permissions: Run the command as a root user: Syntax Example Change the file permissions: Example Start ceph-mon again: Example Additional Resources The Ceph Monitor store is getting too big Ceph Monitor is out of quorum 4.6. Opening port for Ceph manager The ceph-mgr daemons receive placement group information from OSDs on the same range of ports as the ceph-osd daemons. If these ports are not open, a cluster will devolve from HEALTH_OK to HEALTH_WARN and will indicate that PGs are unknown with a percentage count of the PGs unknown. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to Ceph manager. Procedure To resolve this situation, for each host running ceph-mgr daemons, open ports 6800-7300 . Example Restart the ceph-mgr daemons. 4.7. Recovering the Ceph Monitor for bare-metal deployments If all the monitors are down in your Red Hat Ceph Storage cluster, and the ceph -s command does not execute as expected, you can recover the monitors using the monmaptool command. The monmaptool command rebuilds the Ceph monitor store from the keyring files of the daemons. Note This procedure is for bare-metal Red Hat Ceph Storage deployments only. For containerized Red Hat Ceph Storage deployments, see the Knowledgebase article MON recovery procedure for Red Hat Ceph Storage containerized deployment when all the three mon are down. . Prerequisites Bare-metal deployed Red Hat Ceph Storage cluster. Root-level access to all the nodes. All the Ceph monitors are down. Procedure Log into the monitor node. From the monitor nodes, if you are unable to access the OSD nodes without being the root user, then copy the public key pair to the OSD nodes: Generate the SSH key pair, accept the default file name, and leave the passphrase empty: Example Copy the public key to all the OSD nodes in the storage cluster: Example Stop the OSD daemon service on all the OSD nodes: Example To collect the cluster map from all the OSD nodes, create the recovery file and execute the script: Create the recovery file: Example Add the following content to the file and replace the OSD_NODES with either the IP addresses of all the OSD nodes or the hostname of the all OSD nodes in the Red Hat Ceph Storage cluster: Syntax Provide executable permission on the file: Example Execute the file to gather the keyrings of all the OSDs from all the OSD nodes in the storage cluster: Example Fetch the keyrings of other daemons from the respective nodes: For Ceph Monitor, the keyring is the same for all the Ceph monitors. Syntax Example For Ceph Manager, fetch the keyring from all the manager nodes: Syntax Example For Ceph OSDs, the keyrings are generated from the above script and stored in a temporary path: In this example, the OSD keyrings are stored in the /tmp/monstore/keyring file. For the client, fetch the keyring from all the client nodes: Syntax Example For meta data server (MDS), fetch the keyring from all the Ceph MDS nodes: Syntax Example For this key ring append the following caps if not exist: For Ceph Object Gateway, fetch the keyring from all the Ceph Object Gateway nodes: Syntax Example For this key ring append the following caps if not exist: On the Ansible administration node, create a file with all the keyrings fetched from the step: Example Optional: On each Ceph Monitor node, ensure that the monitor map is not available: Example From each Ceph Monitor node, fetch the MONITOR_ID , IP_ADDRESS_OF_MONITOR , and FSID from the etc/ceph/ceph.conf file: Example On the Ceph Monitor node, rebuild the monitor map: Syntax Example On the Ceph Monitor node, check the generated monitor map: Syntax Example On the Ceph Monitor node where we are recovering the monitors, rebuild the Ceph Monitor store from the collected map: Syntax In this example, the recovery is run on the mons-1 node. Example Change the ownership of monstore directory to ceph: Example On all the Ceph Monitor nodes, take a back-up of the corrupted store: Example On all the Ceph Monitor nodes, replace the corrupted store: Example On all the Ceph Monitor nodes, change the owner of the new store: Example On all the Ceph OSD nodes, start the OSDs: Example On all the Ceph Monitor nodes, start the monitors Example 4.8. Recovering the Ceph Monitor store Ceph Monitors store the cluster map in a key-value store such as LevelDB. If the store is corrupted on a Monitor, the Monitor terminates unexpectedly and fails to start again. The Ceph logs might include the following errors: Production Red Hat Ceph Storage clusters use at least three Ceph Monitors so that if one fails, it can be replaced with another one. However, under certain circumstances, all Ceph Monitors can have corrupted stores. For example, when the Ceph Monitor nodes have incorrectly configured disk or file system settings, a power outage can corrupt the underlying file system. If there is corruption on all Ceph Monitors, you can recover it with information stored on the OSD nodes by using utilities called ceph-monstore-tool and ceph-objectstore-tool . Important These procedures cannot recover the following information: Metadata Daemon Server (MDS) keyrings and maps Placement Group settings: full ratio set by using the ceph pg set_full_ratio command nearfull ratio set by using the ceph pg set_nearfull_ratio command Important Never restore the Ceph Monitor store from an old backup. Rebuild the Ceph Monitor store from the current cluster state using the following steps and restore from that. 4.8.1. Recovering the Ceph Monitor store when using BlueStore Follow this procedure if the Ceph Monitor store is corrupted on all Ceph Monitors and you use the BlueStore back end. In containerized environments, this method requires attaching Ceph repositories and restoring to a non-containerized Ceph Monitor first. Warning This procedure can cause data loss. If you are unsure about any step in this procedure, contact the Red Hat Technical Support for an assistance with the recovering process. Prerequisites Bare-metal deployments The rsync and ceph-test packages are installed. Container deployments All OSDs containers are stopped. Enable Ceph repositories on the Ceph nodes based on their roles. The ceph-test and rsync packages are installed on the OSD and Monitor nodes. The ceph-mon package is installed on the Monitor nodes. The ceph-osd package is installed on the OSD nodes. Procedure If you use Ceph in containers , mount all disk with Ceph data to a temporary location. Repeat this step for all OSD nodes. List the data partitions. Use ceph-volume or ceph-disk depending on which utility you used to set up the devices: or Mount the data partitions to temporary location: Restore the SELinux context: Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node. Change the owner and group to ceph:ceph : Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node. Important Due to a bug that causes the update-mon-db command to use additional db and db.slow directories for the Monitor database, you must also copy these directories. To do so: Prepare a temporary location outside the container to mount and access the OSD database and extract the OSD maps needed to restore the Ceph Monitor: Replace OSD-DATA with the Volume Group (VG) or Logical Volume (LV) path to the to the OSD data and OSD-ID with the ID of the OSD. Create a symbolic link between the BlueStore database and block.db : Replace BLUESTORE-DATABASE with the Volume Group (VG) or Logical Volume (LV) path to the BlueStore database and OSD-ID with the ID of the OSD. Use the following commands from the Ceph Monitor node with the corrupted store. Repeat them for all OSDs on all nodes. Collect the cluster map from all OSD nodes: Set the appropriate capabilities: Move all sst file from the db and db.slow directories to the temporary location: Rebuild the Monitor store from the collected map: Note After using this command, only keyrings extracted from the OSDs and the keyring specified on the ceph-monstore-tool command line are present in Ceph's authentication database. You have to recreate or import all other keyrings, such as clients, Ceph Manager, Ceph Object Gateway, and others, so those clients can access the cluster. Back up the corrupted store. Repeat this step for all Ceph Monitor nodes: Replace HOSTNAME with the host name of the Ceph Monitor node. Replace the corrupted store. Repeat this step for all Ceph Monitor nodes: Replace HOSTNAME with the host name of the Monitor node. Change the owner of the new store. Repeat this step for all Ceph Monitor nodes: Replace HOSTNAME with the host name of the Ceph Monitor node. If you use Ceph in containers , then unmount all the temporary mounted OSDs on all nodes: Start all the Ceph Monitor daemons: Ensure that the Monitors are able to form a quorum: Bare-metal deployments Containers Replace HOSTNAME with the host name of the Ceph Monitor node. Import the Ceph Manager keyring and start all Ceph Manager processes: Replace HOSTNAME with the host name of the Ceph Manager node. Start all OSD processes across all OSD nodes: Ensure that the OSDs are returning to service: Bare-metal deployments Containers Replace HOSTNAME with the host name of the Ceph Monitor node. Additional Resources For details on registering Ceph nodes to the Content Delivery Network (CDN), see Registering Red Hat Ceph Storage Nodes to the CDN and Attaching Subscriptions section in the Red Hat Ceph Storage Installation Guide . For details on enabling repositories, see the Enabling the Red Hat Ceph Storage Repositories section in the Red Hat Ceph Storage Installation Guide . 4.9. Additional Resources See Chapter 3, Troubleshooting networking issues in the Red Hat Ceph Storage Troubleshooting Guide for network-related problems.

Commands:
HEALTH_WARN 1 mons down, quorum 1,2 mon.b,mon.c mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
systemctl status ceph-mon@ HOST_NAME systemctl start ceph-mon@ HOST_NAME
Corruption: error in middle of record Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb
Caught signal (Bus error)
ceph daemon ID mon_status
ceph daemon mon.a mon_status
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum) mon.a addr 127.0.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s)
2015-06-04 07:28:32.035795 7f806062e700 0 log [WRN] : mon.a 127.0.0.1:6789/0 clock skew 0.14s > max 0.05s 2015-06-04 04:31:25.773235 7f4997663700 0 log [WRN] : message from mon.1 was stamped 0.186257s in the future, clocks not synchronized
mon.ceph1 store is getting too big! 48031 MB >= 15360 MB -- 62% avail
du -sch /var/lib/ceph/mon/ CLUSTER_NAME - SHORT_HOST_NAME /store.db
du -sch /var/lib/ceph/mon/ceph-host1/store.db 47G /var/lib/ceph/mon/ceph-ceph1/store.db/ 47G total
{ "name": "mon.3", "rank": 2, "state": "peon", "election_epoch": 96, "quorum": [ 1, 2 ], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 1, "fsid": "d5552d32-9d1d-436c-8db1-ab5fc2c63cd0", "modified": "0.000000", "created": "0.000000", "mons": [ { "rank": 0, "name": "mon.1", "addr": "172.25.1.10:6789\/0" }, { "rank": 1, "name": "mon.2", "addr": "172.25.1.12:6789\/0" }, { "rank": 2, "name": "mon.3", "addr": "172.25.1.13:6789\/0" } ] } }
ceph mon getmap -o /tmp/monmap
systemctl stop ceph-mon@<host-name>
systemctl stop ceph-mon@host1
ceph-mon -i ID --extract-monmap /tmp/monmap
ceph-mon -i mon.a --extract-monmap /tmp/monmap
systemctl stop ceph-mon@ HOST_NAME
systemctl stop ceph-mon@host2
su - ceph -c 'ceph-mon -i ID --inject-monmap /tmp/monmap'
su - ceph -c 'ceph-mon -i mon.c --inject-monmap /tmp/monmap'
ceph-mon -i ID --inject-monmap /tmp/monmap
ceph-mon -i mon.c --inject-monmap /tmp/monmap
chown -R ceph:ceph /var/lib/ceph/mon/ceph-c/
systemctl start ceph-mon@host2
systemctl start ceph-mon@host1
rm -rf /var/lib/ceph/mon/ CLUSTER_NAME - SHORT_HOST_NAME
rm -rf /var/lib/ceph/mon/remote-host1
ceph mon remove SHORT_HOST_NAME --cluster CLUSTER_NAME
ceph mon remove host1 --cluster remote
/usr/share/ceph-ansible/ansible-playbook site.yml
ceph tell mon. HOST_NAME compact
ceph tell mon.host1 compact
[mon] mon_compact_on_start = true
systemctl restart ceph-mon@_HOST_NAME_
systemctl restart ceph-mon@host1
ceph mon stat
systemctl status ceph-mon@ HOST_NAME systemctl stop ceph-mon@ HOST_NAME
systemctl status ceph-mon@host1 systemctl stop ceph-mon@host1
su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon. HOST_NAME compact'
su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact'
ceph-monstore-tool /var/lib/ceph/mon/mon. HOST_NAME compact
ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact
chown -R ceph:ceph /var/lib/ceph/mon/mon.node1
systemctl start ceph-mon@ HOST_NAME
systemctl start ceph-mon@host1
firewall-cmd --add-port 6800-7300/tcp firewall-cmd --add-port 6800-7300/tcp --permanent
ssh-keygen
ssh-copy-id root@osds-1 ssh-copy-id root@osds-2 ssh-copy-id root@osds-3
sudo systemctl stop ceph-osd\*.service ceph-osd.target
touch recover.sh
-------------------------------------------------------------------------- NOTE: The directory names specified by 'ms', 'db', and 'db_slow' must end with a trailing / otherwise rsync will not operate properly. -------------------------------------------------------------------------- ms=/tmp/monstore/ db=/root/db/ db_slow=/root/db.slow/ mkdir -p USDms USDdb USDdb_slow -------------------------------------------------------------------------- NOTE: Replace the contents inside double quotes for 'osd_nodes' below with the list of OSD nodes in the environment. -------------------------------------------------------------------------- osd_nodes=" OSD_NODES_1 OSD_NODES_2 OSD_NODES_3 ..." for osd_node in USDosd_nodes; do echo "Operating on USDosd_node" rsync -avz --delete USDms USDosd_node:USDms rsync -avz --delete USDdb USDosd_node:USDdb rsync -avz --delete USDdb_slow USDosd_node:USDdb_slow ssh -t USDosd_node <<EOF for osd in /var/lib/ceph/osd/ceph-*; do ceph-objectstore-tool --type bluestore --data-path \USDosd --op update-mon-db --no-mon-config --mon-store-path USDms if [ -e \USDosd/keyring ]; then cat \USDosd/keyring >> USDms/keyring echo ' caps mgr = "allow profile osd"' >> USDms/keyring echo ' caps mon = "allow profile osd"' >> USDms/keyring echo ' caps osd = "allow *"' >> USDms/keyring else echo WARNING: \USDosd on USDosd_node does not have a local keyring. fi done EOF rsync -avz --delete --remove-source-files USDosd_node:USDms USDms rsync -avz --delete --remove-source-files USDosd_node:USDdb USDdb rsync -avz --delete --remove-source-files USDosd_node:USDdb_slow USDdb_slow done -------------------------------------------------------------------------- End of script ## --------------------------------------------------------------------------
chmod 755 recover.sh
./recovery.sh
cat /var/lib/ceph/mon/ceph- MONITOR_NODE /keyring
cat /var/lib/ceph/mon/ceph-mons-1/keyring
cat /var/lib/ceph/mgr/ceph- MANAGER_NODE /keyring
cat /var/lib/ceph/mgr/ceph-mons-1/keyring
cat /etc/ceph/ CLIENT_KEYRING
cat /etc/ceph/ceph.client.admin.keyring
cat /var/lib/ceph/mds/ceph- MDS_NODE /keyring
cat /var/lib/ceph/mds/ceph-mds-1/keyring
caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *"
cat /var/lib/ceph/radosgw/ceph- CEPH_OBJECT_GATEWAY_NODE /keyring
cat /var/lib/ceph/radosgw/ceph-rgw-1/keyring
caps mon = "allow rw" caps osd = "allow *"
cat /tmp/monstore/keyring [mon.] key = AQAa+RxhAAAAABAApmwud0GQHX0raMBc9zTAYQ== caps mon = "allow *" [client.admin] key = AQAo+RxhcYWtGBAAiY4kKltMGnAXqPLM2A+B8w== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mgr.mons-1] key = AQA++RxhAAAAABAAKdG1ETTEMR8KPa9ZpfcIzw== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.mons-2] key = AQA9+RxhAAAAABAAcCBxsoaIl0sdHTz3dqX4SQ== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.mons-3] key = AQA/+RxhAAAAABAAwe/mwv0hS79fWP+00W6ypQ== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [osd.1] key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.5] key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.8] key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.2] key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.4] key = AQCE+Rxh0mDxDRAApAeqKOJycW5bpP3IuAhSMw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.7] key = AQCJ+Rxhn+RAIhAAp1ImK1jiazBsDpmTQvVEVw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.0] key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.3] key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.6] key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.1] key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.5] key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.8] key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.2] key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.0] key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.3] key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [osd.6] key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" [mds.mds-1] key = AQDs+RxhAF9vERAAdn6ArdUJ31RLr2sBVkzp3A== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [mds.mds-2] key = AQDs+RxhROoAFxAALAgMfM45wC5ht/vSFN2EzQ== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [mds.mds-3] key = AQDs+Rxhd092FRAArXLIHAhMp2z9zcWDCSoIDQ== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [client.rgw.rgws-1.rgw0] key = AQD9+Rxh0iP2MxAAYY76Js1AaZhzFG44cvcyOw== caps mon = "allow rw" caps osd = "allow *"
ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap monmaptool /tmp/monmap --print monmaptool: monmap file /tmp/monmap monmaptool: couldn't open /tmp/monmap: (2) No such file or directory Notice theNo such file or directory error message if monmap is missed Notice that the "No such file or directory" error message if monmap is missed
[global] cluster network = 10.0.208.0/22 fsid = 9877bde8-ccb2-4758-89c3-90ca9550ffea mon host = [v2:10.0.211.00:3300,v1:10.0.211.00:6789],[v2:10.0.211.13:3300,v1:10.0.211.13:6789],[v2:10.0.210.13:3300,v1:10.0.210.13:6789] mon initial members = ceph-mons-1, ceph-mons-2, ceph-mons-3
monmaptool --create --addv MONITOR_ID IP_ADDRESS_OF_MONITOR --enable-all-features --clobber PATH_OF_MONITOR_MAP --fsid FSID
monmaptool --create --addv mons-1 [v2:10.74.177.30:3300,v1:10.74.177.30:6789] --addv mons-2 [v2:10.74.179.197:3300,v1:10.74.179.197:6789] --addv mons-3 [v2:10.74.182.123:3300,v1:10.74.182.123:6789] --enable-all-features --clobber /root/monmap.mons-1 --fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8 monmaptool: monmap file /root/monmap.mons-1 monmaptool: set fsid to 6c01cb34-33bf-44d0-9aec-3432276f6be8 monmaptool: writing epoch 0 to /root/monmap.mon-a (3 monitors)
monmaptool PATH_OF_MONITOR_MAP --print
monmaptool /root/monmap.mons-1 --print monmaptool: monmap file /root/monmap.mons-1 epoch 0 fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8 last_changed 2021-11-23 02:57:23.235505 created 2021-11-23 02:57:23.235505 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.74.177.30:3300/0,v1:10.74.177.30:6789/0] mon.mons-1 1: [v2:10.74.179.197:3300/0,v1:10.74.179.197:6789/0] mon.mons-2 2: [v2:10.74.182.123:3300/0,v1:10.74.182.123:6789/0] mon.mons-3
ceph-monstore-tool /tmp/monstore rebuild -- --keyring KEYRING_PATH --monmap PATH_OF_MONITOR_MAP
ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/monstore/keyring --monmap /root/monmap.mons-1
chown -R ceph:ceph /tmp/monstore
mv /var/lib/ceph/mon/ceph-mons-1/store.db /var/lib/ceph/mon/ceph-mons-1/store.db.corrupted
scp -r /tmp/monstore/store.db mons-1:/var/lib/ceph/mon/ceph-mons-1/
chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db
sudo systemctl start ceph-osd.target
sudo systemctl start ceph-mon.target
Corruption: error in middle of record Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb
ceph-volume lvm list
ceph-disk list
mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-USDi
for i in { OSD_ID }; do restorecon /var/lib/ceph/osd/ceph-USDi; done
for i in { OSD_ID }; do chown -R ceph:ceph /var/lib/ceph/osd/ceph-USDi; done
ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev OSD-DATA --path /var/lib/ceph/osd/ceph- OSD-ID
ln -snf BLUESTORE DATABASE /var/lib/ceph/osd/ceph- OSD-ID /block.db
cd /root/ ms=/tmp/monstore/ db=/root/db/ db_slow=/root/db.slow/ mkdir USDms for host in USDosd_nodes; do echo "USDhost" rsync -avz USDms USDhost:USDms rsync -avz USDdb USDhost:USDdb rsync -avz USDdb_slow USDhost:USDdb_slow rm -rf USDms rm -rf USDdb rm -rf USDdb_slow sh -t USDhost <<EOF for osd in /var/lib/ceph/osd/ceph-*; do ceph-objectstore-tool --type bluestore --data-path \USDosd --op update-mon-db --mon-store-path USDms done EOF rsync -avz USDhost:USDms USDms rsync -avz USDhost:USDdb USDdb rsync -avz USDhost:USDdb_slow USDdb_slow done
ceph-authtool /etc/ceph/ceph.client.admin.keyring -n mon. --cap mon 'allow *' --gen-key cat /etc/ceph/ceph.client.admin.keyring [mon.] key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA== caps mon = "allow *" [client.admin] key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
mv /root/db/*.sst /root/db.slow/*.sst /tmp/monstore/store.db
ceph-monstore-tool /tmp/monstore rebuild -- --keyring /etc/ceph/ceph.client.admin
mv /var/lib/ceph/mon/ceph- HOSTNAME /store.db /var/lib/ceph/mon/ceph- HOSTNAME /store.db.corrupted
scp -r /tmp/monstore/store.db HOSTNAME :/var/lib/ceph/mon/ceph- HOSTNAME /
chown -R ceph:ceph /var/lib/ceph/mon/ceph- HOSTNAME /store.db
umount /var/lib/ceph/osd/ceph-*
systemctl start ceph-mon *
ceph -s
[user@admin ~]USD docker exec ceph-mon-_HOSTNAME_ ceph -s
ceph auth import -i /etc/ceph/ceph.mgr. HOSTNAME .keyring systemctl start ceph-mgr@ HOSTNAME
systemctl start ceph-osd *
ceph -s
[user@admin ~]USD docker exec ceph-mon-_HOSTNAME_ ceph -s