Chapter 5. Upgrading to Red Hat Gluster Storage 3.5 This chapter describes the procedure to upgrade RHGS 3.5 with same RHEL platform major version. To upgrade from RHEL 6 based RHGS to RHEL 7 based RHGS, refer to Upgrading Red Hat Gluster Storage Important Red Hat Enterprise Linux 8 (RHEL 8) is supported only for new installations of Red Hat Gluster Storage 3.5.2 and above. Upgrades to RHEL 8 based Red Hat Gluster Storage are supported from Red Hat Gluster Storage 3.5.2 onwards. Upgrade is not supported from RHEL 7 based RHGS cluster to RHEL 8 based RHGS cluster. Upgrade support limitations While upgrading RHGS from versions lower than RHGS-3.5.4 to version RHGS-3.5.4 or higher, both servers and clients must upgrade to RHGS-3.5.4 or higher versions before bumping up the operating version of the cluster. Virtual Data Optimizer (VDO) volumes, which are supported in Red Hat Enterprise Linux 7.5, are not currently supported in Red Hat Gluster Storage. VDO is supported only when used as part of Red Hat Hyperconverged Infrastructure for Virtualization 2.0. See Understanding VDO for more information. Servers must be upgraded prior to upgrading clients. If you are upgrading from Red Hat Gluster Storage 3.1 Update 2 or earlier, you must upgrade servers and clients simultaneously. If you use NFS-Ganesha, your supported upgrade path to Red Hat Gluster Storage 3.5 depends on the version from which you are upgrading. If you are upgrading from version 3.3 or earlier, use Offline Upgrade to Red Hat Gluster Storage 3.3 and then perform an in-service upgrade from version 3.3 to 3.4. Later, perform the upgrading from version 3.4 to 3.5 using Section 5.2, "In-Service Software Upgrade from Red Hat Gluster Storage 3.4 to Red Hat Gluster Storage 3.5" . If you are upgrading from version 3.4 to 3.5, directly use Section 5.2, "In-Service Software Upgrade from Red Hat Gluster Storage 3.4 to Red Hat Gluster Storage 3.5" . 5.1. Offline Upgrade to Red Hat Gluster Storage 3.5 Warning Before you upgrade, be aware of changed requirements that exist after Red Hat Gluster Storage 3.1.3. If you want to access a volume being provided by a Red Hat Gluster Storage 3.1.3 or higher server, your client must also be using Red Hat Gluster Storage 3.1.3 or higher. Accessing volumes from other client versions can result in data becoming unavailable and problems with directory operations. This requirement exists because Red Hat Gluster Storage 3.1.3 contained a number of changes that affect how the Distributed Hash Table works in order to improve directory consistency and remove the effects seen in BZ#1115367 and BZ#1118762 . Important In Red Hat Enterprise Linux 7 based Red Hat Gluster Storage 3.1 and higher, updating reloads firewall rules. All runtime-only changes made before the reload are lost, so ensure that any changes you want to keep are made persistently. 5.1.1. Upgrading to Red Hat Gluster Storage 3.5 for Systems Subscribed to Red Hat Subscription Manager Procedure 5.1. Before you upgrade Back up the following configuration directory and files in a location that is not on the operating system partition. /var/lib/glusterd /etc/samba /etc/ctdb /etc/glusterfs /var/lib/samba /var/lib/ctdb /var/run/gluster/shared_storage/nfs-ganesha Note With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ . If you use NFS-Ganesha, back up the following files from all nodes: /run/gluster/shared_storage/nfs-ganesha/exports/export.*.conf /etc/ganesha/ganesha.conf /etc/ganesha/ganesha-ha.conf If upgrading from Red Hat Gluster Storage 3.3 to 3.4 or subsequent releases, back up all xattr by executing the following command individually on the brick root(s) for all nodes: Unmount gluster volumes from all clients. On a client, use the following command to unmount a volume from a mount point. If you use NFS-Ganesha, run the following on a gluster server to disable the nfs-ganesha service: On a gluster server, disable the shared volume. Stop all volumes Verify that all volumes are stopped. Stop the glusterd services on all servers using the following command: Stop the pcsd service. Procedure 5.2. Upgrade using yum Note Verify that your system is not on the legacy Red Hat Network Classic update system. If you are still on Red Hat Network Classic, run the following command to migrate to Red Hat Subscription Manager. Then verify that your status has changed. If you use Samba: For Red Hat Enterprise Linux 6.7 or higher, enable the following repository: For Red Hat Enterprise Linux 7, enable the following repository: For Red Hat Enterprise Linux 8, enable the following repository: Ensure that Samba is upgraded on all the nodes simultaneously, as running different versions of Samba in the same cluster will lead to data corruption. Stop the CTDB and SMB services. On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8: On Red Hat Enterprise Linux 6: To verify that services are stopped, run: Upgrade the server to Red Hat Gluster Storage 3.5. Wait for the update to complete. Important Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 7: Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 8: If you use Samba/CTDB, update the following files to replace META="all" with META="<ctdb_volume_name>" , for example, META="ctdb" : /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh - This script ensures the file system and its lock volume are mounted on all Red Hat Gluster Storage servers that use Samba, and ensures that CTDB starts at system boot. /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh - This script ensures that the file system and its lock volume are unmounted when the CTDB volume is stopped. Note For RHEL based Red Hat Gluster Storage upgrading to 3.5 batch update 4 with Samba, the write-behind translator has to manually disabled for all existing samba volumes. Reboot the server to ensure that kernel updates are applied. Ensure that glusterd and pcsd services are started. Note During upgrade of servers, the glustershd.log file throws some "Invalid argument" errors during every index crawl (10 mins by default) on the upgraded nodes. It is *EXPECTED* and can be *IGNORED* until the op-version bump up, after which these errors are not triggered. Sample error message: If you are in op-version '70000' or lower, do not bump up the op-version to '70100' or higher until all the servers and clients are upgraded to the newer version. When all nodes have been upgraded, run the following command to update the op-version of the cluster. This helps to prevent any compatibility issues within the cluster. Note 70200 is the cluster.op-version value for Red Hat Gluster Storage 3.5. Ater upgrading the cluster-op version, enable the granular-entry-heal for the volume via the given command: The feature is now enabled by default post upgrade to Red Hat Gluster Storage 3.5, but this will come into affect only after bumping up the op-version. Refer to Section 1.5, "Red Hat Gluster Storage Software Components and Versions" for the correct cluster.op-version value for other versions. Important If the op-version is bumped up to '70100' after upgrading the servers and before upgrading the clients, some internal metadata files under the root of the mount point named '.glusterfs-anonymous-inode-(gfid)' exposed to the older clients. The clients must not do any I/O or remove or touch contents in this directory. The clients must upgrade to 3.5.4 or higher version, then this directory becomes invisible to the clients. If you want to migrate from Gluster NFS to NFS Ganesha as part of this upgrade, install the NFS-Ganesha packages as described in Chapter 4, Deploying NFS-Ganesha on Red Hat Gluster Storage , and configure the NFS Ganesha cluster using the information in the NFS Ganesha section of the Red Hat Gluster Storage 3.5 Administration Guide . Start all volumes. If you are using NFS-Ganesha: Copy the volume's export information from your backup copy of ganesha.conf to the new /etc/ganesha/ganesha.conf file. The export information in the backed up file is similar to the following: Note With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ . Copy the backup volume export files from the backup directory to /etc/ganesha/exports by running the following command from the backup directory: Enable the shared volume. Ensure that the shared storage volume is mounted on the server. If the volume is not mounted, run the following command: Ensure that the /var/run/gluster/shared_storage/nfs-ganesha directory is created. Enable firewall settings for new services and ports. See Getting Started in the Red Hat Gluster Storage 3.5 Administration Guide . If you use Samba/CTDB: Mount /gluster/lock before starting CTDB by executing the following commands: Verify that the lock volume mounted correctly by checking for lock in the output of the mount command on any Samba server. If all servers that host volumes accessed via SMB have been updated, then start the CTDB and Samba services by executing the following commands. On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8: On Red Hat Enterprise Linux 6: To verify that the CTDB and SMB services have started, execute the following command: If you use NFS-Ganesha: Copy the ganesha.conf and ganesha-ha.conf files, and the /etc/ganesha/exports directory to the /var/run/gluster/shared_storage/nfs-ganesha directory. Update the path of any export entries in the ganesha.conf file. Run the following to clean up any existing cluster configuration: If you have upgraded to Red Hat Enterprise Linux 7.4 or later, set the following SELinux Boolean: Start the nfs-ganesha service and verify that all nodes are functional. Enable NFS-Ganesha on all volumes. 5.1.2. Upgrading to Red Hat Gluster Storage 3.5 for Systems Subscribed to Red Hat Network Satellite Server Procedure 5.3. Before you upgrade Back up the following configuration directory and files in a location that is not on the operating system partition. /var/lib/glusterd /etc/samba /etc/ctdb /etc/glusterfs /var/lib/samba /var/lib/ctdb /var/run/gluster/shared_storage/nfs-ganesha Note With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ . If you use NFS-Ganesha, back up the following files from all nodes: /run/gluster/shared_storage/nfs-ganesha/exports/export.*.conf /etc/ganesha/ganesha.conf /etc/ganesha/ganesha-ha.conf If upgrading from Red Hat Gluster Storage 3.3 to 3.4 or subsequent releases, back up all xattr by executing the following command individually on the brick root(s) for all nodes: Unmount gluster volumes from all clients. On a client, use the following command to unmount a volume from a mount point. If you use NFS-Ganesha, run the following on a gluster server to disable the nfs-ganesha service: On a gluster server, disable the shared volume. Stop all volumes. Verify that all volumes are stopped. Stop the glusterd services on all servers using the following command: Stop the pcsd service. Procedure 5.4. Upgrade using Satellite Create an Activation Key at the Red Hat Network Satellite Server, and associate it with the following channels. For more information, see Section 2.6, "Installing from Red Hat Satellite Server" For Red Hat Enterprise Linux 6.7 or higher: If you use Samba, add the following channel: For Red Hat Enterprise Linux 7: If you use Samba, add the following channel: Unregister your system from Red Hat Network Satellite by following these steps: Log in to the Red Hat Network Satellite server. Click on the Systems tab in the top navigation bar and then the name of the old or duplicated system in the System List . Click the delete system link in the top-right corner of the page. To confirm the system profile deletion by clicking the Delete System button. Run the following command on your Red Hat Gluster Storage server, using your credentials and the Activation Key you prepared earlier. This re-registers the system to the Red Hat Gluster Storage 3.5 channels on the Red Hat Network Satellite Server. Verify that the channel subscriptions have been updated. On Red Hat Enterprise Linux 6.7 and higher, look for the following channels, as well as the rh-gluster-3-samba-for-rhel-6-server-rpms channel if you use Samba. On Red Hat Enterprise Linux 7, look for the following channels, as well as the rh-gluster-3-samba-for-rhel-7-server-rpms channel if you use Samba. Upgrade to Red Hat Gluster Storage 3.5. Important Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 7: Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 8: Reboot the server and run volume and data integrity checks. When all nodes have been upgraded, run the following command to update the op-version of the cluster. This helps to prevent any compatibility issues within the cluster. Note 70200 is the cluster.op-version value for Red Hat Gluster Storage 3.5. Ater upgrading the cluster-op version, enable the granular-entry-heal for the volume via the given command: The feature is now enabled by default post upgrade to Red Hat Gluster Storage 3.5, but this will come into affect only after bumping up the op-version. Refer to Section 1.5, "Red Hat Gluster Storage Software Components and Versions" for the correct cluster.op-version value for other versions. Start all volumes. If you are using NFS-Ganesha: Copy the volume's export information from your backup copy of ganesha.conf to the new /etc/ganesha/ganesha.conf file. The export information in the backed up file is similar to the following: Note With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ . Copy the backup volume export files from the backup directory to /etc/ganesha/exports by running the following command from the backup directory: Enable the shared volume. Ensure that the shared storage volume is mounted on the server. If the volume is not mounted, run the following command: Ensure that the /var/run/gluster/shared_storage/nfs-ganesha directory is created. Enable firewall settings for new services and ports. See Getting Started in the Red Hat Gluster Storage 3.5 Administration Guide . If you use Samba/CTDB: Mount /gluster/lock before starting CTDB by executing the following commands: Verify that the lock volume mounted correctly by checking for lock in the output of the mount command on any Samba server. If all servers that host volumes accessed via SMB have been updated, then start the CTDB and Samba services by executing the following commands. On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8: On Red Hat Enterprise Linux 6: To verify that the CTDB and SMB services have started, execute the following command: If you use NFS-Ganesha: Copy the ganesha.conf and ganesha-ha.conf files, and the /etc/ganesha/exports directory to the /var/run/gluster/shared_storage/nfs-ganesha directory. Update the path of any export entries in the ganesha.conf file. Run the following to clean up any existing cluster configuration: If you have upgraded to Red Hat Enterprise Linux 7.4 or later, set the following SELinux Boolean: Start the ctdb service (and nfs-ganesha service, if used) and verify that all nodes are functional. If this deployment uses NFS-Ganesha, enable NFS-Ganesha on all volumes. 5.1.3. Special consideration for Offline Software Upgrade 5.1.3.1. Migrate CTDB configuration files If you are upgrading CTDB to 4.9.x and beyond from an older version of CTDB, perform the following steps post the upgrade: Make a temporary directory to migrate configuration files. Run the CTDB configuration migration script. The script assumes that the CTDB configuration directory is /etc/ctdb . If this is not correct for your setup, specify an alternative configuration directory with the -d option, for example: Verify that the /tmp/ctdb-migration directory now contains the following files: commands.sh ctdb.conf script.options ctdb.tunables (if additional changes are required) ctdb.sysconfig (if additional changes are required) README.warn (if additional changes are required) Back up the current configuration files. Install the new configuration files. Make the commands.sh file executable, and run it. If /tmp/ctdb-migration /ctdb.tunables exists, move it to the /etc/ctdb directory. If /tmp/ctdb-migration /ctdb.sysconfig exists, back up the old /etc/sysconfig/ctdb file and replace it with /tmp/ctdb-migration /ctdb.sysconfig . Otherwise, back up the old /etc/sysconfig/ctdb file and replace it with /etc/sysconfig/ctdb.rpmnew .

Commands:
find ./ -type d ! -path "./.*" ! -path "./" | xargs getfattr -d -m. -e hex > /var/log/glusterfs/xattr_dump_ brick_name
umount mount-point
gluster nfs-ganesha disable
gluster volume set all cluster.enable-shared-storage disable
for vol in `gluster volume list`; do gluster --mode=script volume stop USDvol; sleep 2s; done
gluster volume info
service glusterd stop pkill glusterfs pkill glusterfsd
systemctl stop pcsd
migrate-rhs-classic-to-rhsm --status
migrate-rhs-classic-to-rhsm --rhn-to-rhsm
migrate-rhs-classic-to-rhsm --status
subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-6-server-rpms
subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-7-server-rpms
subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-8-x86_64-rpms
systemctl stop ctdb
service ctdb stop
ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'
yum update
yum install nfs-ganesha-selinux
dnf install glusterfs-ganesha
gluster volume set <volname> performance.write-behind off
systemctl start glusterd systemctl start pcsd
[2021-05-25 17:58:38.007134] E [MSGID: 114031] [client-rpc-fops_v2.c:216:client4_0_mkdir_cbk] 0-spvol-client-40: remote operation failed. Path: (null) [Invalid argument]
gluster volume set all cluster.op-version 70200
gluster volume heal USDVOLNAME granular-entry-heal enable
for vol in `gluster volume list`; do gluster --mode=script volume start USDvol; sleep 2s; done
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v1.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v2.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v3.conf"
cp export.* /etc/ganesha/exports/
gluster volume set all cluster.enable-shared-storage enable
mount -t glusterfs hostname :gluster_shared_storage /var/run/gluster/shared_storage
cd /var/run/gluster/shared_storage/ mkdir nfs-ganesha
mount <ctdb_volume_name> mount -t glusterfs server:/ctdb_volume_name /gluster/lock/
mount | grep 'lock' <hostname>:/<ctdb_volume_name>.tcp on /gluster/lock type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
systemctl start ctdb
service ctdb start
ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'
cd /etc/ganesha/ cp ganesha.conf ganesha-ha.conf /var/run/gluster/shared_storage/nfs-ganesha/ cp -r exports/ /var/run/gluster/shared_storage/nfs-ganesha/
sed -i 's/\/etc\/ganesha/\/var\/run\/gluster\/shared_storage\/nfs-ganesha/' /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf
/usr/libexec/ganesha/ganesha-ha.sh --cleanup /var/run/gluster/shared_storage/nfs-ganesha
setsebool -P ganesha_use_fusefs on
gluster nfs-ganesha enable
gluster volume set volname ganesha.enable on
find ./ -type d ! -path "./.*" ! -path "./" | xargs getfattr -d -m. -e hex > /var/log/glusterfs/xattr_dump_ brick_name
umount mount-point
gluster nfs-ganesha disable
gluster volume set all cluster.enable-shared-storage disable
for vol in `gluster volume list`; do gluster --mode=script volume stop USDvol; sleep 2s; done
gluster volume info
service glusterd stop pkill glusterfs pkill glusterfsd
systemctl stop pcsd
Base Channel: Red Hat Enterprise Linux Server (v.6 for 64-bit x86_64) Child channels: RHEL Server Scalable File System (v. 6 for x86_64) Red Hat Gluster Storage Server 3 (RHEL 6 for x86_64)
Red Hat Gluster 3 Samba (RHEL 6 for x86_64)
Base Channel: Red Hat Enterprise Linux Server (v.7 for 64-bit x86_64) Child channels: RHEL Server Scalable File System (v. 7 for x86_64) Red Hat Gluster Storage Server 3 (RHEL 7 for x86_64)
Red Hat Gluster 3 Samba (RHEL 7 for x86_64)
rhnreg_ks --username username --password password --force --activationkey Activation Key ID
rhn-channel --list rhel-6-server-rpms rhel-scalefs-for-rhel-6-server-rpms rhs-3-for-rhel-6-server-rpms
rhn-channel --list rhel-7-server-rpms rh-gluster-3-for-rhel-7-server-rpms
yum update
yum install nfs-ganesha-selinux
dnf install glusterfs-ganesha
gluster volume set all cluster.op-version 70200
gluster volume heal USDVOLNAME granular-entry-heal enable
for vol in `gluster volume list`; do gluster --mode=script volume start USDvol; sleep 2s; done
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v1.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v2.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v3.conf"
cp export.* /etc/ganesha/exports/
gluster volume set all cluster.enable-shared-storage enable
mount -t glusterfs hostname :gluster_shared_storage /var/run/gluster/shared_storage
cd /var/run/gluster/shared_storage/ mkdir nfs-ganesha
mount <ctdb_volume_name> mount -t glusterfs server:/ctdb_volume_name /gluster/lock/
mount | grep 'lock' <hostname>:/<ctdb_volume_name>.tcp on /gluster/lock type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
systemctl start ctdb
service ctdb start
ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'
cd /etc/ganesha/ cp ganesha.conf ganesha-ha.conf /var/run/gluster/shared_storage/nfs-ganesha/ cp -r exports/ /var/run/gluster/shared_storage/nfs-ganesha/
sed -i 's/\/etc\/ganesha/\/var\/run\/gluster\/shared_storage\/nfs-ganesha/' /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf
/usr/libexec/ganesha/ganesha-ha.sh --cleanup /var/run/gluster/shared_storage/nfs-ganesha
setsebool -P ganesha_use_fusefs on
systemctl start ctdb gluster nfs-ganesha enable
gluster volume set volname ganesha.enable on
mkdir /tmp/ctdb-migration
./usr/share/doc/ctdb-4.9. x /examples/config_migrate.sh -o /tmp/ctdb-migration /etc/sysconfig/ctdb
./usr/share/doc/ctdb-4.9. x /examples/config_migrate.sh -o /tmp/ctdb-migration /etc/sysconfig/ctdb -d ctdb-config-dir
mv /etc/ctdb/ctdb.conf /etc/ctdb/ctdb.conf.default
mv /tmp/ctdb-migration /ctdb.conf /etc/ctdb/ctdb.conf mv script.options /etc/ctdb/
chmod +x /tmp/ctdb-migration /commands.sh . /tmp/ctdb-migration /commands.sh
cp /tmp/ctdb-migration /ctdb.tunables /etc/ctdb
mv /etc/sysconfig/ctdb /etc/sysconfig/ctdb.old mv /tmp/ctdb-migration /ctdb.sysconfig /etc/sysconfig/ctdb
mv /etc/sysconfig/ctdb /etc/sysconfig/ctdb.old mv /etc/sysconfig/ctdb.rpmnew /etc/sysconfig/ctdb