11.11. Rebalancing Volumes If a volume has been expanded or shrunk using the add-brick or remove-brick commands, the data on the volume needs to be rebalanced among the servers. Note In a non-replicated volume, all bricks should be online to perform the rebalance operation using the start option. In a replicated volume, at least one of the bricks in the replica should be online. To rebalance a volume, use the following command on any of the servers: For example: When run without the force option, the rebalance command attempts to balance the space utilized across nodes. Files whose migration would cause the target node to have less available space than the source node are skipped. This results in linkto files being retained, which may cause slower access when a large number of linkto files are present. Red Hat strongly recommends you to disconnect all the older clients before executing the rebalance command to avoid a potential data loss scenario. Warning The Rebalance command can be executed with the force option even when the older clients are connected to the cluster. However, this could lead to a data loss situation. A rebalance operation with force , balances the data based on the layout, and hence optimizes or does away with the link files, but may lead to an imbalanced storage space used across bricks. This option is to be used only when there are a large number of link files in the system. To rebalance a volume forcefully, use the following command on any of the servers: For example: 11.11.1. Rebalance Throttling The rebalance process uses multiple threads to ensure good performance during migration of multiple files. During multiple file migration, there can be a severe impact on storage system performance and a throttling mechanism is provided to manage it. By default, the rebalance throttling is started in the normal mode. Configure the throttling modes to adjust the rate at which the files must be migrated For example: 11.11.2. Displaying Rebalance Progress To display the status of a volume rebalance operation, use the following command: For example: A rebalance operation starts a rebalance process on each node of the volume. Each process is responsible for rebalancing the files on its own individual node. Each row of the rebalance status output describes the progress of the operation on a single node. Important If there is a reboot while rebalancing the rebalance output might display an incorrect status and some files might not get rebalanced. Workaround: After the reboot, once the rebalance is completed, trigger another rebalance so that the files that were not balanced during the reboot are now rebalanced correctly, and the rebalance output gives the correct status. The following table describes the output of the rebalance status command: Table 11.2. Rebalance Status Output Description Property Name Description Node The name of the node. Rebalanced-files The number of files that were successfully migrated. size The total size of the files that were migrated. scanned The number of files scanned on the node. This includes the files that were migrated. failures The number of files that could not be migrated because of errors. skipped The number of files which were skipped because of various errors or reasons. status The status of the rebalance operation on the node is in progress , completed , or failed . run time in h:m:s The amount of time for which the process has been running on the node. The estimated time left for the rebalance to complete on all nodes is also displayed. The estimated time to complete is displayed only after the rebalance operation has been running for 10 minutes. In cases where the remaining time is extremely large, the estimated time to completion is displayed as >2 months and the user is advised to check again later. The time taken to complete a rebalance operation depends on the number of files estimated to be on the bricks and the rate at which files are being processed by the rebalance process. This value is recalculated every time the rebalance status command is executed and becomes more accurate the longer rebalance has been running, and for large data sets. The calculation assumes that a file system partition contains a single brick. A rebalance balance operation is considered complete when the status of every node is completed . For example: With this release, details about the files that are skipped during rebalance operation can be obtained. Entries of all such files are available in the rebalance log with the message ID 109126. You can search for the message ID from the log file and get the list of all the skipped files: For example: To know more about the failed files, search for 'migrate-data failed' in the rebalance.log file. However, the count for rebalance failed files will not match with "migrate-data failed" in the rebalance.log because the failed count includes all possible failures and just not file migration. 11.11.3. Stopping a Rebalance Operation To stop a rebalance operation, use the following command: For example:

Commands:
gluster volume rebalance VOLNAME start
gluster volume rebalance test-volume start Starting rebalancing on volume test-volume has been successful
gluster volume rebalance VOLNAME start force
gluster volume rebalance test-volume start force Starting rebalancing on volume test-volume has been successful
gluster volume set VOLNAME rebal-throttle lazy|normal|aggressive
gluster volume set test-volume rebal-throttle lazy
gluster volume rebalance VOLNAME status
gluster volume rebalance test-volume status Node Rebalanced size scanned failures skipped status run time -files in h:m:s ------------- ---------- ------ ------- -------- ------- ----------- -------- localhost 71962 70.3GB 380852 0 0 in progress 2:02:20 server1 70489 68.8GB 502185 0 0 in progress 2:02:20 server2 70704 69.0GB 507728 0 0 in progress 2:02:20 server3 71819 70.1GB 435611 0 0 in progress 2:02:20 Estimated time left for rebalance to complete : 2:50:24
gluster volume rebalance test-volume status Node Rebalanced size scanned failures skipped status run time -files in h:m:s ---------- ---------- ----- ------- -------- ------- ----------- -------- node2 0 0Bytes 0 0 0 completed 0:02:23 node3 234 737.8KB 3350 0 257 completed 0:02:25 node4 3 14.6K 71 0 6 completed 0:00:02 localhost 317 1.1MB 3484 0 155 completed 0:02:38 volume rebalance: test-volume: success
grep -i 109126 /var/log/glusterfs/test-volume-rebalance.log [2018-03-15 09:14:30.203393] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-fs-orangefs. [2018-03-15 09:14:31.262969] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-devices. [2018-03-15 09:14:31.842631] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/stable/sysfs-devices-system-cpu. [2018-03-15 09:14:33.733728] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/testing/sysfs-bus-fcoe. [2018-03-15 09:14:35.576404] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/ABI/testing/sysfs-bus-iio-frequency-ad9523. [2018-03-15 09:14:43.378480] I [MSGID: 109126] [dht-rebalance.c:2715:gf_defrag_migrate_single_file] 0-test-volume-dht: File migration skipped for /linux-4.9.27/Documentation/DocBook/kgdb.tmpl.
gluster volume rebalance VOLNAME stop
gluster volume rebalance test-volume stop Node Rebalanced size scanned failures skipped status run time -files in h:m:s ------------- ---------- ------- ------- -------- ------- ----------- -------- localhost 106504 104.0GB 558111 0 0 stopped 3:02:24 server1 102299 99.9GB 725239 0 0 stopped 3:02:24 server2 102264 99.9GB 737364 0 0 stopped 3:02:24 server3 106813 104.3GB 646581 0 0 stopped 3:02:24 Estimated time left for rebalance to complete : 2:06:38