elasticsearch/elasticsearch — Issue #1731: Zen discovery stops working after a ConnectException
Author: nkvoll
Created: 2012-02-23 10:24:22+00:00
State: open

In 0.19.0-RC3, I started a two-node cluster with ``minimum_master_nodes: 2``, waited for the cluster state to become green then shut down one of the nodes. This showed up in the logs for the remaining node:

[2012-02-23 11:09:21,023][INFO ][discovery.zen ] [Malus, Karl] master_left [[Nitro][VUh02v-qSp2F-0m0wbDIDA][inet[/192.168.1.147:9303]]], reason [shut_down]
[2012-02-23 11:09:21,024][WARN ][discovery.zen ] [Malus, Karl] not enough master nodes after master left (reason = shut_down), current nodes: {[Malus, Karl][wN5y7pQ_TYCLi2ywrcTSjw][inet[192.168.1.147/192.168.1.147:9304]],}
[2012-02-23 11:09:21,025][INFO ][cluster.service ] [Malus, Karl] removed {[Nitro][VUh02v-qSp2F-0m0wbDIDA][inet[/192.168.1.147:9303]],}, reason: zen-disco-master_failed ([Nitro][VUh02v-qSp2F-0m0wbDIDA][inet[/192.168.1.147:9303]])
[2012-02-23 11:09:24,035][WARN ][discovery.zen ] [Malus, Karl] failed to connect to master [[Nitro][VUh02v-qSp2F-0m0wbDIDA][inet[/192.168.1.147:9303]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Nitro][inet[/192.168.1.147:9303]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:562)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:505)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:484)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(ZenDiscovery.java:312)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:69)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:266)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:400)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:362)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:284)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more

At this point, zen discovery stopped working. I restarted the other node, but they cannot find/connect to eachother properly. The node that was not shut down raises MasterNotDiscoveredExceptions, while the restarted selects itself as the master and permanently has cluster state: red.

I cannot _consistently_ reproduce this issue, but I'm thinking maybe this exception should be handled?