Redis cluster configuration by example
By: Mohammed Hewedy
Tags: Redis, Clustering, Sharding

High Available clustered Redis

This configuration is the same as the previous one, except we add a minimum of 1 replica node for each master one for each master server. so the minimum configuration here is 6 nodes. so we now have High availability and automatic failover (will see in the example section below).

The HA clustered Redis uses similar configuration as the clustered Redis, the only difference is that we need to start 6 nodes as a minimum and when we start the cluster we use the flag --cluster-replicas 1 , let’s see an example creating command:

Example:

First, create the configurations for 6 machines:

vermin@verminbox:~$ for i in 0 1 2 3 4 5

do

p=$((7000 + i))

mkdir -p node_$p

cat << EOF > node_$p/redis.conf

port $p

cluster-enabled yes

cluster-config-file cluster.conf

cluster-node-timeout 5000

appendonly yes

EOF

done

Second, start the 6 machines:

vermin@verminbox:~$ for i in 0 1 2 3 4 5; do cd node_$((7000 + i)); /usr/bin/redis-server redis.conf& cd -; done

Third, let’s start the cluster on the 6 machines (need to type yes when asked):

vermin@verminbox:~$ redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \

127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \

--cluster-replicas 1

This command creates a cluster of 3 master and 3 replicas.

Testing automatic failover

First, let’s check our cluster using CLUSTER NODES :

vermin@verminbox:~$ redis-cli -c -p 7000

127.0.0.1:7000> CLUSTER NODES 39a53d804303f855c9df28c0c286cb95d26b8da4 127.0.0.1:7004@17004 slave 2bc439369730811542be11864e7ae0928ff681d9 0 1590523800617 5 connected

f90e8ecf684407d1b7902d4de5b524e7766dbeac 127.0.0.1:7005@17005 slave 4067752947767d3a41f6362158bf37c0edb59f21 0 1590523799000 6 connected

2bc439369730811542be11864e7ae0928ff681d9 127.0.0.1:7001@17001 master - 0 1590523800517 2 connected 5461-10922

42ccd0590dd8afd2fcf0047208305d0d44772d98 127.0.0.1:7000@17000 myself,master - 0 1590523798000 1 connected 0-5460

4067752947767d3a41f6362158bf37c0edb59f21 127.0.0.1:7002@17002 master - 0 1590523799611 3 connected 10923-16383

2fdd597c4e4d9a6c6be2351a2a06535d7445ef10 127.0.0.1:7003@17003 slave 42ccd0590dd8afd2fcf0047208305d0d44772d98 0 1590523800000 4 connected

It shows that the node at port 7000 , 7001 and 7002 are master, so let’s kill node at 7001 it and see what will happen:

vermin@verminbox:~$ kill -9 $(ps aux | grep [7]001 | cut -d " " -f6 )

We will notice that the following will be printed:

17083:S 26 May 2020 20:11:42.211 # Connection with master lost.

17083:S 26 May 2020 20:11:42.211 * Caching the disconnected master state.

17083:S 26 May 2020 20:11:42.483 * Connecting to MASTER 127.0.0.1:7001

17083:S 26 May 2020 20:11:42.484 * MASTER <-> REPLICA sync started

17083:S 26 May 2020 20:11:42.484 # Error condition on socket for SYNC: Connection refused

......

......

17082:S 26 May 2020 20:11:47.522 * Marking node 2bc439369730811542be11864e7ae0928ff681d9 as failing (quorum reached).

.......

17083:S 26 May 2020 20:11:47.930 # Cluster state changed: fail

17083:S 26 May 2020 20:11:47.931 # Start of election delayed for 737 milliseconds (rank #0, offset 154).

.........

17081:M 26 May 2020 20:11:48.741 # Failover auth granted to 39a53d804303f855c9df28c0c286cb95d26b8da4 for epoch 7

17079:M 26 May 2020 20:11:48.741 # Failover auth granted to 39a53d804303f855c9df28c0c286cb95d26b8da4 for epoch 7

17083:S 26 May 2020 20:11:48.752 # Failover election won: I'm the new master.

.......

Now let’s test the cluster from the client, we will notice the 3 masters are still running, which means the :