Chapter 7. Replacing a non-primary host using new bricks 7.1. Host replacement prerequisites Determine which node to use as the Ansible controller node (the node from which all Ansible playbooks are executed). Red Hat recommends using a healthy node in the same cluster as the failed node as the Ansible controller node. Stop brick processes and unmount file systems on the failed host, to avoid file system inconsistency issues. Check which operating system is running on your hyperconverged hosts by running the following command: Install the same operating system on a replacement host. 7.2. Preparing the cluster for host replacement Verify host state in the Administrator Portal. Log in to the Red Hat Virtualization Administrator Portal. The host is listed as NonResponsive in the Administrator Portal. Virtual machines that previously ran on this host are in the Unknown state. Click Compute Hosts and click the Action menu (...). Click Confirm host has been rebooted and confirm the operation. Verify that the virtual machines are now listed with a state of Down . Update the SSH fingerprint for the failed node. Log in to the Ansible controller node as the root user. Remove the existing SSH fingerprint for the failed node. Copy the public key from the Ansible controller node to the freshly installed node. Verify that you can log in to all hosts in the cluster, including the Ansible controller node, using key-based SSH authentication without a password. Test access using all network addresses. The following example assumes that the Ansible controller node is host1 . Use ssh-copy-id to copy the public key to any host you cannot log into without a password using this method. 7.3. Creating the node_prep_inventory.yml file Define the replacement node in the node_prep_inventory.yml file. Procedure Familiarize yourself with your Gluster configuration. The configuration that you define in your inventory file must match the existing Gluster volume configuration. Use gluster volume info to check where your bricks should be mounted for each Gluster volume, for example: Back up the node_prep_inventory.yml file. Edit the node_prep_inventory.yml file to define your node preparation. See Appendix B, Understanding the node_prep_inventory.yml file for more information about this inventory file and its parameters. 7.4. Creating the node_replace_inventory.yml file Define your cluster hosts by creating a node_replacement_inventory.yml file. Procedure Back up the node_replace_inventory.yml file. Edit the node_replace_inventory.yml file to define your cluster. See Appendix C, Understanding the node_replace_inventory.yml file for more information about this inventory file and its parameters. 7.5. Executing the replace_node.yml playbook file The replace_node.yml playbook reconfigures a Red Hat Hyperconverged Infrastructure for Virtualization cluster to use a new node after an existing cluster node has failed. Procedure Execute the playbook. 7.6. Removing a failed host from the cluster When a replacement host is ready, remove the existing failed host from the cluster. Procedure Remove the failed host. Log in to the Administrator Portal. Click Compute Hosts . The replacement host is in the NonResponsive state. Virtual machines running on that host are in the Unknown state. Select the replacement host. Click the main Action menu (...) for the Hosts page and select Confirm host has been rebooted . Click OK to confirm. Click the Action menu (...) beside the failed host and click Remove . Clean stale Hosted Engine metadata. Determine the identifier of the failed node. Remove the metadata associated with that host identifier. 7.7. Verifying healing in progress After replacing a failed host with a new host, verify that your storage is healing as expected. Procedure Verify that healing is in progress. Run the following command on any hyperconverged host: The output shows a summary of healing activity on each brick in each volume, for example: Depending on brick size, volumes can take a long time to heal. You can still run and migrate virtual machines using this node while the underlying storage heals.

Commands:
pkill glusterfsd umount /gluster_bricks/{engine,vmstore,data}
nodectl info
sed -i `/ failed-host-frontend.example.com /d` /root/.ssh/known_hosts sed -i `/ failed-host-backend.example.com /d` /root/.ssh/known_hosts
ssh-copy-id root@ new-host-backend.example.com ssh-copy-id root@ new-host-frontend.example.com
ssh root@ host1-backend.example.com ssh root@ host1-frontend.example.com ssh root@ host2-backend.example.com ssh root@ host2-frontend.example.com ssh root@ new-host-backend.example.com ssh root@ new-host-frontend.example.com
ssh-copy-id root@ host-frontend.example.com ssh-copy-id root@ host-backend.example.com
gluster volume info engine | grep -i brick Number of Bricks: 1 x 3 = 3 Bricks: Brick1: host1.example.com:/gluster_bricks/engine/engine Brick2: host2.example.com:/gluster_bricks/engine/engine Brick3: host3.example.com:/gluster_bricks/engine/engine
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment cp node_prep_inventory.yml node_prep_inventory.yml.bk
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment cp node_replace_inventory.yml node_replace_inventory.yml.bk
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/ ansible-playbook -i node_prep_inventory.yml -i node_replace_inventory.yml tasks/replace_node.yml
hosted-engine --vm-status | grep failed-node.example.com --== Host server1-frontend.example.com ( id: 1 ) status ==-- Hostname : failed-node.example.com
hosted-engine --clean-metadata --host-id=1 --force
for vol in `gluster volume list`; do gluster volume heal USDvol info summary; done
Brick brick1 Status: Connected Total Number of entries: 3 Number of entries in heal pending: 2 Number of entries in split-brain: 1 Number of entries possibly healing: 0