xetorthio/jedis — pull request #564 by HeartSaVioR (2014-02-25 11:44:24+00:00)
Title: Let Jedis Cluster support failover (implementation changed more likely to antirez's redis-rb-cluster)

This PR is started from Google Groups - https://groups.google.com/d/msg/jedis_redis/HtDblX771bs/rI2gDiRVGqYJ

Jedis Cluster could not take care of failover, but ruby client did it.
I seeked ruby redis clients that supports cluster, and found antirez's implementation!(https://github.com/antirez/redis-rb-cluster)

Maybe @marcosnils would referenced it, because method naming is same. :)

I found difference of two implementation, and finally found how redis-rb-cluster take care of failover.
If one of node is down, it just retry with random & alive node, and when random node respond to MOVED, it switches slot so node will not referenced to this slot.
(Maybe any nodes would reply CLUSTERDOWN while node is down but failover doesn't end yet.
So we need to retry several times - same to JedisSentinelPool.)

There is some more differences on two implementation, so I ported differences to current Jedis Cluster.
(Changelog is included in commit log, and below.)

I have not included unit test to kill node and watch failover.
(It needs "cluster replicate" command to configure cluster environment.
And it's pain to implement force kill node and cleanup!
We need additional 6 cluster nodes with configuring "mutual exclusive" failover test.)

But I've checked failover with code snippet and configure.

A. Code Snippet

````
package net.heartsavior.dev;

import java.util.HashSet;
import java.util.Set;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException;
import redis.clients.jedis.exceptions.JedisConnectionException;
import redis.clients.jedis.exceptions.JedisDataException;

public class JedisClusterTest {

public static void main(String[] args) {
Set jedisClusterNodes = new HashSet();

//Jedis Cluster will attempt to discover cluster nodes automatically
jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7379));

JedisCluster jc = new JedisCluster(jedisClusterNodes);
System.out.println("Currently " + jc.getClusterNodes().size() + " nodes in cluster");

jc.set("foo", "bar");
String value = jc.get("foo");
System.out.println(value);

jc.set("loop", String.valueOf(0));

while (true) {
try {
long currCount = jc.incr("loop");
System.out.println("Current loop count : " + currCount);
} catch (JedisConnectionException e) {
System.out.println("JedisConnectionException!! - exc: " + e);
e.printStackTrace();
} catch (JedisClusterMaxRedirectionsException e) {
System.out.println("JedisClusterMaxRedirectionsException!! - exc: " + e);
e.printStackTrace();
} catch (JedisDataException e) {
System.out.println("JedisDataException!! - exc: " + e);
e.printStackTrace();
}finally {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// pass
}
}
}
}
}
````

B. Cluster Configure

* 6 nodes, 7379 ~ 7384
* redis-trib.rb create --replicas 1 127.0.0.1:7379 127.0.0.1:7380 127.0.0.1:7381 127.0.0.1:7382 127.0.0.1:7383 127.0.0.1:7384
* it automatically configure cluster with 7379(M) 7381(S) / 7380(M) 7382(S) / 7381(M) 7383(S)

C. Output

C1. start running

````
Currently 6 nodes in cluster
bar
Current loop count : 1
Current loop count : 2
Current loop count : 3
Current loop count : 4
Current loop count : 5
Current loop count : 6
Current loop count : 7
Current loop count : 8
Current loop count : 9
Current loop count : 10
Current loop count : 11
Current loop count : 12
Current loop count : 13
Current loop count : 14
Current loop count : 15
Current loop count : 16
Current loop count : 17
Current loop count : 18
Current loop count : 19
Current loop count : 20
Current loop count : 21
````

C2. Killing 7379 (echo "debug segfault" | redis-cli -p 7379)

````
JedisDataException!! - exc: redis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information
redis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information
at redis.clients.jedis.Protocol.processError(Protocol.java:100)
at redis.clients.jedis.Protocol.process(Protocol.java:118)
at redis.clients.jedis.Protocol.read(Protocol.java:187)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:201)
at redis.clients.jedis.Jedis.incr(Jedis.java:575)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:260)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:1)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:61)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:72)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:33)
at redis.clients.jedis.JedisCluster.incr(JedisCluster.java:262)
at net.heartsavior.dev.JedisClusterTest.main(JedisClusterTest.java:31)
Current loop count : 22
...
Current loop count : 31
````

C3. Killing 7380

````
JedisDataException!! - exc: redis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more informationredis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information

at redis.clients.jedis.Protocol.processError(Protocol.java:100)
at redis.clients.jedis.Protocol.process(Protocol.java:118)
at redis.clients.jedis.Protocol.read(Protocol.java:187)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:201)
at redis.clients.jedis.Jedis.incr(Jedis.java:575)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:260)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:1)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:61)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:33)
at redis.clients.jedis.JedisCluster.incr(JedisCluster.java:262)
at net.heartsavior.dev.JedisClusterTest.main(JedisClusterTest.java:31)
... (some other CLUSTERDOWN errors)

Current loop count : 32
...
Current loop count : 48
````

C4. Killing 7381

````
JedisDataException!! - exc: redis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information
redis.clients.jedis.exceptions.JedisDataException: CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information
at redis.clients.jedis.Protocol.processError(Protocol.java:100)
at redis.clients.jedis.Protocol.process(Protocol.java:118)
at redis.clients.jedis.Protocol.read(Protocol.java:187)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:201)
at redis.clients.jedis.Jedis.incr(Jedis.java:575)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:260)
at redis.clients.jedis.JedisCluster$20.execute(JedisCluster.java:1)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:61)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:33)
at redis.clients.jedis.JedisCluster.incr(JedisCluster.java:262)
at net.heartsavior.dev.JedisClusterTest.main(JedisClusterTest.java:31)
... (some other CLUSTERDOWN errors)
Current loop count : 49
...
````

I will try to make unit test after introducing "cluster replicate" command.

I've not checked multi-threaded environment, so I wish to lend a hand if you have multi-threaded cluster test codes. ;)

I've not implemented some of point from antirez's thing.
(He always re-initialize slots cache and discover new nodes when MOVED occurred.)
It could make multi-thread problem because node / slot is not thread-safe.
But there's some pros : if node is down, antirez's implementation re-initialize all slot caches so MOVED occurred just once, and it could be easy to discover new nodes attached later.
So we can choose we include this functionality and have node / slot being synchronized, or don't include and make Jedis simple.

Please review and comment!
Thanks!

Below is change log
----

JedisClusterCommand

* improvements on connection error handling
** if based on slot connection throws connection related exception, retry to random node
** if we retry with random node, but all nodes are unreachable, throw JedisConnectionException without retry
** try to release connection whether connection is broken or not

* bug fix : if asking flag is on, and success this time, set asking flag to off

JedisClusterConnectionHandler

* have flexibility on initializing slots cache
** allow some nodes connection failure - skip
** if current node is success initializing slots cache, skip other nodes
** if current node failed to initialize slots cache, discard all discovered nodes and slots

* set nodes if node does not exist in nodes
** it restricts JedisPool to replace - prevent IllegalStateException : Returned object not currently part of this pool

JedisSlotBasedConnectionGuaranteedConnectionHandler

* getConnection (random connection)
** check all connections by random sequence
** always return valid connection (able to ping-pong)
** throw exception if all connections are invalid

* some refactoring