Chapter 6. Troubleshooting Ceph MDSs As a storage administrator, you can troubleshoot the most common issues that can occur when using the Ceph Metadata Server (MDS). Some of the common errors that you might encounter: An MDS node failure requiring a new MDS deployment. An MDS node issue requiring redeployment of an MDS node. 6.1. Redeploying a Ceph MDS Ceph Metadata Server (MDS) daemons are necessary for deploying a Ceph File System. If an MDS node in your cluster fails, you can redeploy a Ceph Metadata Server by removing an MDS server and adding a new or existing server. You can use the command-line interface or Ansible playbook to add or remove an MDS server. 6.1.1. Prerequisites A running Red Hat Ceph Storage cluster. 6.1.2. Removing a Ceph MDS using Ansible To remove a Ceph Metadata Server (MDS) using Ansible, use the shrink-mds playbook. Note If there is no replacement MDS to take over once the MDS is removed, the file system will become unavailable to clients. If that is not desirable, consider adding an additional MDS before removing the MDS you would like to take offline. Prerequisites At least one MDS node. A running Red Hat Ceph Storage cluster deployed by Ansible. Root or sudo access to an Ansible administration node. Procedure Log in to the Ansible administration node. Change to the /usr/share/ceph-ansible directory: Example Run the Ansible shrink-mds.yml playbook, and when prompted, type yes to confirm shrinking the cluster: Syntax Replace ID with the ID of the MDS node you want to remove. You can remove only one Ceph MDS each time the playbook runs. Example As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and remove the MDS node under the [mdss] section: Syntax Example In this example, node02 was removed from the [mdss] list. Verification Check the status of the MDS daemons: Syntax Example Additional Resources For more information on installing Red Hat Ceph Storage, see the Red Hat Ceph Storage Installation Guide . See the Adding a Ceph MDS using Ansible section in the Red Hat Ceph Storage Troubleshooting Guide for more details on adding an MDS using Ansible. 6.1.3. Removing a Ceph MDS using the command-line interface You can manually remove a Ceph Metadata Server (MDS) using the command-line interface. Note If there is no replacement MDS to take over once the current MDS is removed, the file system will become unavailable to clients. If that is not desirable, consider adding an MDS before removing the existing MDS. Prerequisites The ceph-common package is installed. A running Red Hat Ceph Storage cluster. Root or sudo access to the MDS nodes. Procedure Log into the Ceph MDS node that you want to remove the MDS daemon from. Stop the Ceph MDS service: Syntax Replace HOST_NAME with the short name of the host where the daemon is running. Example Disable the MDS service if you are not redeploying MDS to this node: Syntax Replace HOST_NAME with the short name of the host to disable the daemon. Example Remove the /var/lib/ceph/mds/ceph- MDS_ID directory on the MDS node: Syntax Replace MDS_ID with the ID of the MDS node that you want to remove the MDS daemon from. Example Verification Check the status of the MDS daemons: Syntax Example Additional Resources For more information on installing Red Hat Ceph Storage, see the Red Hat Ceph Storage Installation Guide . See the Adding a Ceph MDS using the command line interface section in the Red Hat Ceph Storage Troubleshooting Guide for more details on adding an MDS using the command line interface. 6.1.4. Adding a Ceph MDS using Ansible Use the Ansible playbook to add a Ceph Metadata Server (MDS). Prerequisites A running Red Hat Ceph Storage cluster deployed by Ansible. Root or sudo access to an Ansible administration node. New or existing servers that can be provisioned as MDS nodes. Procedure Log in to the Ansible administration node Change to the /usr/share/ceph-ansible directory: Example As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and add the MDS node under the [mdss] section: Syntax Replace NEW_MDS_NODE_NAME with the host name of the node where you want to install the MDS server. Alternatively, you can colocate the MDS daemon with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections. Example As the ansible user, run the Ansible playbook to provision the MDS node: Bare-metal deployments: Container deployments: After the Ansible playbook has finished running, the new Ceph MDS node appears in the storage cluster. Verification Check the status of the MDS daemons: Syntax Example Alternatively, you can use the ceph mds stat command to check if the MDS is in an active state: Syntax Example Additional Resources For more information on installing Red Hat Ceph Storage, see the Red Hat Ceph Storage Installation Guide . See the Removing a Ceph MDS using Ansible section in the Red Hat Ceph Storage Troubleshooting Guide for more details on removing an MDS using Ansible. 6.1.5. Adding a Ceph MDS using the command-line interface You can manually add a Ceph Metadata Server (MDS) using the command-line interface. Prerequisites The ceph-common package is installed. A running Red Hat Ceph Storage cluster. Root or sudo access to the MDS nodes. New or existing servers that can be provisioned as MDS nodes. Procedure Add a new MDS node by logging into the node and creating an MDS mount point: Syntax Replace MDS_ID with the ID of the MDS node that you want to add the MDS daemon to. Example If this is a new MDS node, create the authentication key if you are using Cephx authentication: Syntax Replace MDS_ID with the ID of the MDS node to deploy the MDS daemon on. Example Note Cephx authentication is enabled by default. See the Cephx authentication link in the Additional Resources section for more information about Cephx authentication. Start the MDS daemon: Syntax Replace HOST_NAME with the short name of the host to start the daemon. Example Enable the MDS service: Syntax Replace HOST_NAME with the short name of the host to enable the service. Example Verification Check the status of the MDS daemons: Syntax Example Alternatively, you can use the ceph mds stat command to check if the MDS is in an active state: Syntax Example Additional Resources For more information on installing Red Hat Ceph Storage, see the Red Hat Ceph Storage Installation Guide . For more information on Cephx authentication, see the Red Hat Ceph Storage Configuration Guide . See the Removing a Ceph MDS using the command line interface section in the Red Hat Ceph Storage Troubleshooting Guide for more details on removing an MDS using the command line interface.

Commands:
[ansible@admin ~]USD cd /usr/share/ceph-ansible
ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill= ID -i hosts
[ansible @admin ceph-ansible]USD ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=node02 -i hosts
[mdss] MDS_NODE_NAME MDS_NODE_NAME
[mdss] node01 node03
ceph fs dump
[ansible@admin ceph-ansible]USD ceph fs dump [mds.node01 {0:115304} state up:active seq 5 addr [v2:172.25.250.10:6800/695510951,v1:172.25.250.10:6801/695510951]] Standby daemons: [mds.node03 {-1:144437} state up:standby seq 2 addr [v2:172.25.250.11:6800/172950087,v1:172.25.250.11:6801/172950087]]
sudo systemctl stop ceph-mds@ HOST_NAME
[admin@node02 ~]USD sudo systemctl stop ceph-mds@node02
sudo systemctl disable ceph-mds@ HOST_NAME
[admin@node02 ~]USD sudo systemctl disable ceph-mds@node02
sudo rm -fr /var/lib/ceph/mds/ceph- MDS_ID
[admin@node02 ~]USD sudo rm -fr /var/lib/ceph/mds/ceph-node02
ceph fs dump
[ansible@admin ceph-ansible]USD ceph fs dump [mds.node01 {0:115304} state up:active seq 5 addr [v2:172.25.250.10:6800/695510951,v1:172.25.250.10:6801/695510951]] Standby daemons: [mds.node03 {-1:144437} state up:standby seq 2 addr [v2:172.25.250.11:6800/172950087,v1:172.25.250.11:6801/172950087]]
[ansible@admin ~]USD cd /usr/share/ceph-ansible
[mdss] MDS_NODE_NAME NEW_MDS_NODE_NAME
[mdss] node01 node03
[ansible@admin ceph-ansible]USD ansible-playbook site.yml --limit mdss -i hosts
[ansible@admin ceph-ansible]USD ansible-playbook site-container.yml --limit mdss -i hosts
ceph fs dump
[ansible@admin ceph-ansible]USD ceph fs dump [mds.node01 {0:115304} state up:active seq 5 addr [v2:172.25.250.10:6800/695510951,v1:172.25.250.10:6801/695510951]] Standby daemons: [mds.node03 {-1:144437} state up:standby seq 2 addr [v2:172.25.250.11:6800/172950087,v1:172.25.250.11:6801/172950087]]
ceph mds stat
[ansible@admin ceph-ansible]USD ceph mds stat cephfs:1 {0=node01=up:active} 1 up:standby
sudo mkdir /var/lib/ceph/mds/ceph- MDS_ID
[admin@node03 ~]USD sudo mkdir /var/lib/ceph/mds/ceph-node03
sudo ceph auth get-or-create mds. MDS_ID mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph- MDS_ID /keyring
[admin@node03 ~]USD sudo ceph auth get-or-create mds.node03 mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-node03/keyring
sudo systemctl start ceph-mds@ HOST_NAME
[admin@node03 ~]USD sudo systemctl start ceph-mds@node03
systemctl enable ceph-mds@ HOST_NAME
[admin@node03 ~]USD sudo systemctl enable ceph-mds@node03
ceph fs dump
[admin@mon]USD ceph fs dump [mds.node01 {0:115304} state up:active seq 5 addr [v2:172.25.250.10:6800/695510951,v1:172.25.250.10:6801/695510951]] Standby daemons: [mds.node03 {-1:144437} state up:standby seq 2 addr [v2:172.25.250.11:6800/172950087,v1:172.25.250.11:6801/172950087]]
ceph mds stat
[ansible@admin ceph-ansible]USD ceph mds stat cephfs:1 {0=node01=up:active} 1 up:standby