4.13. Configuring Red Hat Gluster Storage 4.13.1. Peer the Nodes From rhgs-primary-n01, peer nodes rhgs-primary-n{02..20}: From rhgs-primary-n02, re-peer node rhgs-primary-n01: Note the problem. This step is done in order to clean up the initial peering process which leaves rhgs-primary-n01 defined by its IP address in the other peers Gluster trusted pool configuration files. This is important because the IP addresses are ephemeral: And correct it: From rhgs--n01, peer nodes rhgs-secondary-n{02..10}: From rhgs-secondary-n02, peer node rhgs-secondary-n01: 4.13.2. Creating Distribute-Replicate Volumes Warning Support for two-way replication is planned for deprecation and removal in future versions of Red Hat Gluster Storage. This will affect both replicated and distributed-replicated volumes. Support is being removed because two-way replication does not provide adequate protection from split-brain conditions. While a dummy node can be used as an interim solution for this problem, Red Hat recommends that all volumes that currently use two-way replication are migrated to use either arbitrated replication or three-way replication. Instructions for migrating a two-way replicated volume to an arbitrated replicated volume are available in Converting to an Arbitrated Volume . Information about three-way replication is available in Creating Three-way Replicated Volumes and Creating Three-way Distributed Replicated Volumes . On the primary trusted pool, create a 10x2 Distribute-Replicate volume, ensuring that bricks are paired appropriately with their replica peers as defined in Section 4.1.3, "Primary Storage Pool Configuration" . The resulting Gluster volume topology is: On the secondary trusted pool, create a 10-brick Distribute volume: The resulting Gluster volume topology is: 4.13.3. Setting up Geo-Replication from the Primary to the Secondary Region From a primary region node, establish geo-replication from the local myvol volume to the remote region myvol-slave volume. As a prerequisite, all secondary/slave side nodes must allow root user login via SSH. The below commands should be run on all of nodes rhgs-secondary-n{01..10}. Create an SSH key pair for the root user on rhgs-primary-n01, and copy the contents of the public key: On rhgs-secondary-n01, add the SSH public key from rhgs-primary-n01 to the root user's authorized_keys file: Note The above SSH public key is for illustration purposes only. Use the key from your own id_rsa.pub file on rhgs-primary-n01. At this point, the root user on rhgs-primary-n01 should have passwordless SSH access to rhgs-secondary-n01. This is a prerequisite for setting up geo-replication. Create a common pem pub file on rhgs-primary-n01: Note This must be done on the node where passwordless SSH to the secondary node was configured. Create the geo-replication session from the primary site to the secondary site. The push-pem option is needed to perform the necessary pem-file setup on the slave nodes. Verify the geo-replication status. After a few minutes, the initialization stage should complete, and each connection should show Active or Passive for its status. At this point, the 100 TB Gluster volume is fully ready for use, with cross-zone synchronous data replication on the primary side and remote asynchronous data replication to a read-only volume on the secondary side located in a separate region.

Commands:
for i in {02..20}; do gluster peer probe rhgs-primary-nUSD{i}; done
gluster peer status | grep Hostname | grep -v rhgs Hostname: 10.240.21.133
gluster peer probe rhgs-primary-n01 peer probe: success. gluster peer status | grep Hostname | grep n01 Hostname: rhgs-primary-n01
for i in {02..10}; do gluster peer probe rhgs-secondary-nUSD{i}; done
gluster peer probe rhgs-secondary-n01
gluster volume create myvol replica 2 rhgs-primary-n01:/rhgs/bricks/myvol rhgs-primary-n02:/rhgs/bricks/myvol rhgs-primary-n03:/rhgs/bricks/myvol rhgs-primary-n04:/rhgs/bricks/myvol rhgs-primary-n05:/rhgs/bricks/myvol rhgs-primary-n06:/rhgs/bricks/myvol rhgs-primary-n07:/rhgs/bricks/myvol rhgs-primary-n08:/rhgs/bricks/myvol rhgs-primary-n09:/rhgs/bricks/myvol rhgs-primary-n10:/rhgs/bricks/myvol rhgs-primary-n11:/rhgs/bricks/myvol rhgs-primary-n12:/rhgs/bricks/myvol rhgs-primary-n13:/rhgs/bricks/myvol rhgs-primary-n14:/rhgs/bricks/myvol rhgs-primary-n15:/rhgs/bricks/myvol rhgs-primary-n16:/rhgs/bricks/myvol rhgs-primary-n17:/rhgs/bricks/myvol rhgs-primary-n18:/rhgs/bricks/myvol rhgs-primary-n19:/rhgs/bricks/myvol rhgs-primary-n20:/rhgs/bricks/myvol volume create: myvol: success: please start the volume to access data gluster volume start myvol volume start: myvol: success gluster volume info myvol Volume Name: myvol Type: Distributed-Replicate Volume ID: f093e120-b291-4362-a859-8d2d4dd87f3a Status: Started Snap Volume: no Number of Bricks: 10 x 2 = 20 Transport-type: tcp Bricks: Brick1: rhgs-primary-n01:/rhgs/bricks/myvol Brick2: rhgs-primary-n02:/rhgs/bricks/myvol Brick3: rhgs-primary-n03:/rhgs/bricks/myvol Brick4: rhgs-primary-n04:/rhgs/bricks/myvol Brick5: rhgs-primary-n05:/rhgs/bricks/myvol Brick6: rhgs-primary-n06:/rhgs/bricks/myvol Brick7: rhgs-primary-n07:/rhgs/bricks/myvol Brick8: rhgs-primary-n08:/rhgs/bricks/myvol Brick9: rhgs-primary-n09:/rhgs/bricks/myvol Brick10: rhgs-primary-n10:/rhgs/bricks/myvol Brick11: rhgs-primary-n11:/rhgs/bricks/myvol Brick12: rhgs-primary-n12:/rhgs/bricks/myvol Brick13: rhgs-primary-n13:/rhgs/bricks/myvol Brick14: rhgs-primary-n14:/rhgs/bricks/myvol Brick15: rhgs-primary-n15:/rhgs/bricks/myvol Brick16: rhgs-primary-n16:/rhgs/bricks/myvol Brick17: rhgs-primary-n17:/rhgs/bricks/myvol Brick18: rhgs-primary-n18:/rhgs/bricks/myvol Brick19: rhgs-primary-n19:/rhgs/bricks/myvol Brick20: rhgs-primary-n20:/rhgs/bricks/myvol Options Reconfigured: performance.readdir-ahead: on auto-delete: disable snap-max-soft-limit: 90 snap-max-hard-limit: 256
Distribute set | +-- Replica set 0 | | | +-- Brick 0: rhgs-primary-n01:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n02:/rhgs/bricks/myvol | +-- Replica set 1 | | | +-- Brick 0: rhgs-primary-n03:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n04:/rhgs/bricks/myvol | +-- Replica set 2 | | | +-- Brick 0: rhgs-primary-n05:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n06:/rhgs/bricks/myvol | +-- Replica set 3 | | | +-- Brick 0: rhgs-primary-n07:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n08:/rhgs/bricks/myvol | +-- Replica set 4 | | | +-- Brick 0: rhgs-primary-n09:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n10:/rhgs/bricks/myvol | +-- Replica set 5 | | | +-- Brick 0: rhgs-primary-n11:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n12:/rhgs/bricks/myvol | +-- Replica set 6 | | | +-- Brick 0: rhgs-primary-n13:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n14:/rhgs/bricks/myvol | +-- Replica set 7 | | | +-- Brick 0: rhgs-primary-n15:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n16:/rhgs/bricks/myvol | +-- Replica set 8 | | | +-- Brick 0: rhgs-primary-n17:/rhgs/bricks/myvol | | | +-- Brick 1: rhgs-primary-n18:/rhgs/bricks/myvol | +-- Replica set 9 | +-- Brick 0: rhgs-primary-n19:/rhgs/bricks/myvol | +-- Brick 1: rhgs-primary-n20:/rhgs/bricks/myvol
gluster volume create myvol-slave rhgs-secondary-n01:/rhgs/bricks/myvol-slave rhgs-secondary-n02:/rhgs/bricks/myvol-slave rhgs-secondary-n03:/rhgs/bricks/myvol-slave rhgs-secondary-n04:/rhgs/bricks/myvol-slave rhgs-secondary-n05:/rhgs/bricks/myvol-slave rhgs-secondary-n06:/rhgs/bricks/myvol-slave rhgs-secondary-n07:/rhgs/bricks/myvol-slave rhgs-secondary-n08:/rhgs/bricks/myvol-slave rhgs-secondary-n09:/rhgs/bricks/myvol-slave rhgs-secondary-n10:/rhgs/bricks/myvol-slave volume create: myvol-slave: success: please start the volume to access data
gluster volume start myvol-slave volume start: myvol-slave: success
gluster volume info myvol-slave Volume Name: myvol-slave Type: Distribute Volume ID: 64295b00-ac19-436c-9aac-6069e0a5b8cf Status: Started Snap Volume: no Number of Bricks: 10 Transport-type: tcp Bricks: Brick1: rhgs-secondary-n01:/rhgs/bricks/myvol-slave Brick2: rhgs-secondary-n02:/rhgs/bricks/myvol-slave Brick3: rhgs-secondary-n03:/rhgs/bricks/myvol-slave Brick4: rhgs-secondary-n04:/rhgs/bricks/myvol-slave Brick5: rhgs-secondary-n05:/rhgs/bricks/myvol-slave Brick6: rhgs-secondary-n06:/rhgs/bricks/myvol-slave Brick7: rhgs-secondary-n07:/rhgs/bricks/myvol-slave Brick8: rhgs-secondary-n08:/rhgs/bricks/myvol-slave Brick9: rhgs-secondary-n09:/rhgs/bricks/myvol-slave Brick10: rhgs-secondary-n10:/rhgs/bricks/myvol-slave Options Reconfigured: performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable
Distribute set | +-- Brick 0: rhgs-secondary-n01:/rhgs/bricks/myvol-slave | +-- Brick 1: rhgs-secondary-n02:/rhgs/bricks/myvol-slave | +-- Brick 2: rhgs-secondary-n03:/rhgs/bricks/myvol-slave | +-- Brick 3: rhgs-secondary-n04:/rhgs/bricks/myvol-slave | +-- Brick 4: rhgs-secondary-n05:/rhgs/bricks/myvol-slave | +-- Brick 5: rhgs-secondary-n06:/rhgs/bricks/myvol-slave | +-- Brick 6: rhgs-secondary-n07:/rhgs/bricks/myvol-slave | +-- Brick 7: rhgs-secondary-n08:/rhgs/bricks/myvol-slave | +-- Brick 8: rhgs-secondary-n09:/rhgs/bricks/myvol-slave | +-- Brick 9: rhgs-secondary-n10:/rhgs/bricks/myvol-slave
sed -i s/PermitRootLogin\ no/PermitRootLogin\ yes/ /etc/ssh/sshd_config service sshd restart
ssh-keygen cat ~root/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAmtzZdIR+pEl16LqH0kbGQfA7sTe1iWHhV/x+5zVDb91Z+gzMVdBTBaLyugeoBlxzOeFFnc/7a9TwNSr7YWt/yKZxh+lnqq /9xcWtONUrfvLH4TEWu4dlRwCvXGsdv23lQK0YabaY9hqzshscFtSnQTmzT13LPc9drH+k7lHBu4KjA4igDvX/j41or0weneg1vcqAP9vRyh4xXgtocqBiAqJegBZ5O /QO1ynyJBysp7tIHF7HZuh3sFCxtqEPPsJkVJDiQZ/NqTr3hAqDzmn4USOX3FbSOvomlWa8We6tGb9nfUH6vBQGyKbWk4YOzm6E5oTzuRBGA1vCPmwpwR/cw== root@rhgs-primary-n01
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAmtzZdIR+pEl16LqH0kbGQfA7sTe1iWHhV/x+5zVDb91Z+gzMVdBTBaLyugeoBlxzOeFFnc/7a9TwNSr7YWt /yKZxh+lnqq7/9xcWtONUrfvLH4TEWu4dlRwCvXGsdv23lQK0YabaY9hqzshscFtSnQTmzT13LPc9drH+k7lHBu4KjA4igDvX j41or0weneg1vcqAP9vRyh4xXgtocqBiAqJegBZ5O/QO1ynyJBysp7tIHF7HZuh3sFCxtqEPPsJkVJDiQZ /NqTr3hAqDzmn4USOX3FbSOvomlWa8We6tGb9nfUH6vBQGyKbWk4YOzm6E5oTzuRBGA1vCPmwpwR/cw== root@rhgs-primary-n01" | sudo tee ~root/.ssh /authorized_keys > /dev/null
gluster system:: execute gsec_create
gluster volume geo-replication myvol rhgs-secondary-n01::myvol-slave create push-pem
gluster volume geo-replication myvol rhgs-secondary-n01::myvol-slave start
gluster volume geo-replication myvol rhgs-secondary-n01::myvol-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ rhgs-primary-n01 myvol /rhgs/bricks/myvol root rhgs-secondary-n10::myvol-slave Active N/A Changelog Crawl rhgs-primary-n18 myvol /rhgs/bricks/myvol root rhgs-secondary-n05::myvol-slave Passive N/A N/A rhgs-primary-n06 myvol /rhgs/bricks/myvol root rhgs-secondary-n07::myvol-slave Passive N/A N/A rhgs-primary-n02 myvol /rhgs/bricks/myvol root rhgs-secondary-n02::myvol-slave Passive N/A N/A rhgs-primary-n10 myvol /rhgs/bricks/myvol root rhgs-secondary-n09::myvol-slave Passive N/A N/A rhgs-primary-n14 myvol /rhgs/bricks/myvol root rhgs-secondary-n01::myvol-slave Passive N/A N/A rhgs-primary-n03 myvol /rhgs/bricks/myvol root rhgs-secondary-n03::myvol-slave Active N/A Changelog Crawl rhgs-primary-n09 myvol /rhgs/bricks/myvol root rhgs-secondary-n08::myvol-slave Active N/A Changelog Crawl rhgs-primary-n11 myvol /rhgs/bricks/myvol root rhgs-secondary-n10::myvol-slave Active N/A Changelog Crawl rhgs-primary-n13 myvol /rhgs/bricks/myvol root rhgs-secondary-n03::myvol-slave Active N/A Changelog Crawl rhgs-primary-n19 myvol /rhgs/bricks/myvol root rhgs-secondary-n08::myvol-slave Active N/A Changelog Crawl rhgs-primary-n17 myvol /rhgs/bricks/myvol root rhgs-secondary-n04::myvol-slave Active N/A Changelog Crawl rhgs-primary-n05 myvol /rhgs/bricks/myvol root rhgs-secondary-n06::myvol-slave Active N/A Changelog Crawl rhgs-primary-n15 myvol /rhgs/bricks/myvol root rhgs-secondary-n06::myvol-slave Active N/A Changelog Crawl rhgs-primary-n16 myvol /rhgs/bricks/myvol root rhgs-secondary-n07::myvol-slave Passive N/A N/A rhgs-primary-n07 myvol /rhgs/bricks/myvol root rhgs-secondary-n04::myvol-slave Active N/A Changelog Crawl rhgs-primary-n20 myvol /rhgs/bricks/myvol root rhgs-secondary-n09::myvol-slave Passive N/A N/A rhgs-primary-n12 myvol /rhgs/bricks/myvol root rhgs-secondary-n02::myvol-slave Passive N/A N/A rhgs-primary-n04 myvol /rhgs/bricks/myvol root rhgs-secondary-n01::myvol-slave Passive N/A N/A rhgs-primary-n08 myvol /rhgs/bricks/myvol root rhgs-secondary-n05::myvol-slave Passive N/A N/A