Chapter 4. Reusing bricks and reconstructing existing brick configuration 4.1. Host replacement prerequisites Determine which node to use as the Ansible controller node (the node from which all Ansible playbooks are executed). Red Hat recommends using a healthy node in the same cluster as the failed node as the Ansible controller node. If the failed host used Network-Bound Disk Encryption, ensure that you know the passphrase used for the existing disks. Take note of the disks that comprise the gluster volumes hosted by the server you are replacing. If possible, locate a recent backup or create a new backup of the important files (disk configuration or inventory files). See Backing up important files for details. Stop brick processes and unmount file systems on the failed host, to avoid file system inconsistency issues. Check which operating system is running on your hyperconverged hosts by running the following command: Reinstall the same operating system on the failed hyperconverged host. 4.2. Preparing the cluster for host replacement Verify host state in the Administrator Portal. Log in to the Red Hat Virtualization Administrator Portal. The host is listed as NonResponsive in the Administrator Portal. Virtual machines that previously ran on this host are in the Unknown state. Click Compute Hosts and click the Action menu (...). Click Confirm host has been rebooted and confirm the operation. Verify that the virtual machines are now listed with a state of Down . Update the SSH fingerprint for the failed node. Log in to the Ansible controller node as the root user. Remove the existing SSH fingerprint for the failed node. Copy the public key from the Ansible controller node to the freshly installed node. Verify that you can log in to all hosts in the cluster, including the Ansible controller node, using key-based SSH authentication without a password. Test access using all network addresses. The following example assumes that the Ansible controller node is host1 . Use ssh-copy-id to copy the public key to any host you cannot log into without a password using this method. 4.3. Recreating disk configuration without backups If you do not have backup configuration files available for your cluster, you can recreate configuration using the following sections to ensure you are still able to use existing bricks and their data. 4.3.1. Reconfiguring encryption during host replacement If the failed host used encryption, but you do not have backup encryption configuration available, you need to recreate your encryption configuration when you replace a failed host. Follow these steps to create encryption configuration files on the replacement host to match the other hosts in your existing cluster. Procedure Set new keys and key files. Store the passphrase for the LUKS encrypted disk in a temporary file in the /root directory. If each disk has a separate passphrase, save them separately. Generate new key files. Generate a random key file for each disk. Set appropriate permissions on the new keyfiles. Set the new key for each disk. Verify each device can be opened with its key file. Determine the LUKS UUID for each device. Open each device using its key file and UUID. For example: Configure automatic decryption at boot time. Add a line for each device to the /etc/crypttab file using the following format. For example: Set up Network-Bound Disk Encryption on the root disk. Change into the hc-ansible-deployment directory: Create the inventory file. Make a copy of the luks_tang_inventory.yml file for future reference. Define your configuration in the luks_tang_inventory.yml file. Use the example luks_tang_inventory.yml file to define the details of disk encryption on each host. A complete outline of this file is available in Understanding the luks_tang_inventory.yml file . Encrypt the luks_tang_inventory.yml file and specify a password using ansible-vault . The required variables in luks_tang_inventory.yml include password values, so it is important to encrypt the file to protect the password values. Enter and confirm a new vault password when prompted. Execute the luks_tang_setup.yml playbook with the bindtang tag. Enter the vault password for this file when prompted to start disk encryption configuration. 4.3.2. Reconfiguring deduplication and compression during host replacement If the failed host used deduplication and compression (VDO), but you do not have backup configuration information available, you need to recreate the deduplication and compression configuration when you replace a failed host. Follow these steps to create deduplication and compression configuration files on the replacement host to match the other hosts in your existing cluster. Procedure Copy the /etc/vdoconf.yml file from a healthy node to the replacement node. Edit the indicated values in the /etc/vdoconf.yml file to provide the correct values for your replacement node. Important Be careful when editing this file. Editing this file by hand is supported only when reconstructing deduplication and compression configuration without a backup file. vdo_sd* Change this parameter to match the name of your VDO device. device Specify the VDO device using its by-id path. For normal volumes, this is something like /dev/disk/by-id/scsi-xxx . For encrypted volumes, this is something like /dev/disk/by-id/dm-uuid-CRYPT-LUKS2-xxxxx . For example: Restart the VDO service. 4.3.3. Restoring disk mount configuration If you do not have backup disk mount configuration, you need to recreate your configuration when you replace a host. Follow these steps to reconstruct disk mount configuration. Procedure Scan existing physical volumes, volume groups, and logical volumes. Determine the UUID of each gluster brick. Add a line to the /etc/fstab file for each gluster brick, using the UUID. Volumes that use deduplication and compression need additional mount options, as shown: Create mount directories based on information from volumes. Mount all bricks. Set the required SELinux labels on all brick mount points. 4.4. Creating the node_prep_inventory.yml file Define the replacement node in the node_prep_inventory.yml file. Procedure Familiarize yourself with your Gluster configuration. The configuration that you define in your inventory file must match the existing Gluster volume configuration. Use gluster volume info to check where your bricks should be mounted for each Gluster volume, for example: Back up the node_prep_inventory.yml file. Edit the node_prep_inventory.yml file to define your node preparation. See Appendix B, Understanding the node_prep_inventory.yml file for more information about this inventory file and its parameters. 4.5. Creating the node_replace_inventory.yml file Define your cluster hosts by creating a node_replacement_inventory.yml file. Procedure Back up the node_replace_inventory.yml file. Edit the node_replace_inventory.yml file to define your cluster. See Appendix C, Understanding the node_replace_inventory.yml file for more information about this inventory file and its parameters. 4.6. Executing the replace_node.yml playbook file The replace_node.yml playbook reconfigures a Red Hat Hyperconverged Infrastructure for Virtualization cluster to use a new node after an existing cluster node has failed. Procedure Execute the playbook. 4.7. Finalizing host replacement After you have replaced a failed host with a new host, follow these steps to ensure that the cluster is connected to the new host and properly activated. Procedure Activate the host. Log in to the Red Hat Virtualization Administrator Portal. Click Compute Hosts and observe that the replacement host is listed with a state of Maintenance . Select the host and click Management Activate . Wait for the host to reach the Up state. Attach the gluster network to the host. Click Compute Hosts and select the host. Click Network Interfaces Setup Host Networks . Drag and drop the newly created network to the correct interface. Ensure that the Verify connectivity between Host and Engine checkbox is checked. Ensure that the Save network configuration checkbox is checked. Click OK to save. Verify the health of the network. Click the Network Interfaces tab and check the state of the host's network. If the network interface enters an "Out of sync" state or does not have an IP Address, click Management Refresh Capabilities . 4.8. Verifying healing in progress After replacing a failed host with a new host, verify that your storage is healing as expected. Procedure Verify that healing is in progress. Run the following command on any hyperconverged host: The output shows a summary of healing activity on each brick in each volume, for example: Depending on brick size, volumes can take a long time to heal. You can still run and migrate virtual machines using this node while the underlying storage heals.

Commands:
pkill glusterfsd umount /gluster_bricks/{engine,vmstore,data}
nodectl info
sed -i `/ failed-host-frontend.example.com /d` /root/.ssh/known_hosts sed -i `/ failed-host-backend.example.com /d` /root/.ssh/known_hosts
ssh-copy-id root@ new-host-backend.example.com ssh-copy-id root@ new-host-frontend.example.com
ssh root@ host1-backend.example.com ssh root@ host1-frontend.example.com ssh root@ host2-backend.example.com ssh root@ host2-frontend.example.com ssh root@ new-host-backend.example.com ssh root@ new-host-frontend.example.com
ssh-copy-id root@ host-frontend.example.com ssh-copy-id root@ host-backend.example.com
echo passphrase /root/key
echo passphraseA /root/sd a _key echo passphraseB /root/sd b _key echo passphraseC /root/sd c _key echo passphraseD /root/sd d _key
for disk in sda sdb sdc sdd; do dd if=/dev/urandom of=/etc/USD{disk}_keyfile bs=1024 count=8192
chown 400 /etc/*_keyfile
cryptsetup luksAddKey /etc/sda_keyfile --key-file /root/sda_key cryptsetup luksAddKey /etc/sdb_keyfile --key-file /root/sdb_key cryptsetup luksAddKey /etc/sdc_keyfile --key-file /root/sdc_key cryptsetup luksAddKey /etc/sdd_keyfile --key-file /root/sdd_key
cryptsetup luksUUID /dev/sd X
cryptsetup luksOpen UUID= sdX-UUID luks_sd X -d /etc/sd X _keyfile
cryptsetup luksOpen UUID=a28a19c7-6028-44df-b0b8-e5245944710c luks_sda -d /etc/sda_keyfile
echo luks_sd X UUID= sdX-UUID /etc/sdX_keyfile >> /etc/crypttab
echo luks_sda UUID=a28a19c7-6028-44df-b0b8-e5245944710c /etc/sda_keyfile >> /etc/crypttab
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment
cp luks_tang_inventory.yml luks_tang_inventory.yml.backup
ansible-vault encrypt luks_tang_inventory.yml
ansible-playbook -i luks_tang_inventory.yml tasks/luks_tang_setup.yml --tags=bindtang --ask-vault-pass
scp /etc/vdoconf.yml root@ new-node.example.com :/etc/
cat /etc/vdoconf.yml config: !Configuration vdos: vdo_sdc : !VDOService device: /dev/disk/by-id/scsi-360030480197f830125618adb17bac04c logicalSize: 180T physicalSize: 18625G
systemctl restart vdo.service
pvscan vgscan lvscan
blkid lv_name
echo "UUID=64dfd1b1-4333-4ef6-8835-1053c6904d93 /gluster_bricks/engine xfs inode64,noatime,nodiratime,_netdev,x-systemd.device-timeout=0 0 0" >> /etc/fstab
echo "UUID=64dfd1b1-4333-4ef6-8835-1053c6904d93 /gluster_bricks/vmstore xfs inode64,noatime,nodiratime,_netdev,x-systemd.device-timeout=0,x-systemd.requires=vdo.service 0 0" >> /etc/fstab
mkdir -p /gluster_bricks/{engine,vmstore,data}
mount -a
semanage fcontext -a -t glusterd_brick_t /gluster_bricks/engine semanage fcontext -a -t glusterd_brick_t /gluster_bricks/vmstore semanage fcontext -a -t glusterd_brick_t /gluster_bricks/data restorecon -Rv /gluster_bricks/engine restorecon -Rv /gluster_bricks/vmstore restorecon -Rv /gluster_bricks/data
gluster volume info engine | grep -i brick Number of Bricks: 1 x 3 = 3 Bricks: Brick1: host1.example.com:/gluster_bricks/engine/engine Brick2: host2.example.com:/gluster_bricks/engine/engine Brick3: host3.example.com:/gluster_bricks/engine/engine
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment cp node_prep_inventory.yml node_prep_inventory.yml.bk
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment cp node_replace_inventory.yml node_replace_inventory.yml.bk
cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/ ansible-playbook -i node_prep_inventory.yml -i node_replace_inventory.yml tasks/replace_node.yml
for vol in `gluster volume list`; do gluster volume heal USDvol info summary; done
Brick brick1 Status: Connected Total Number of entries: 3 Number of entries in heal pending: 2 Number of entries in split-brain: 1 Number of entries possibly healing: 0